Which settings already failed
Data, ML and researchML engineer running training experiments through Claude Code
We were tuning a ranking model over three weeks. Learning rate 1e-3 diverged by step 4,000, and batch size 512 ran out of GPU memory. A week later a new session suggested both again, and we lost an evening finding out twice.
How do I stop Claude Code from suggesting training settings that already failed?
Have the agent record each failed run with ArcRoom's remember_failure: the settings and why, like diverged or out of memory. Before proposing new settings, it checks find_task and gets the failed ones back first, along with settings that worked, saved with complete_task. ArcRoom records each training command and its flags, not logs or loss curves, so only saved results are known.
Also asked as: My agent keeps proposing a learning rate that already diverged. How do I fix that? · How do I keep a record of failed hyperparameter runs that AI agents can read? · Can Claude Code remember which experiment settings ran out of GPU memory?
ArcRoom records each training command with its flags, so the playbook knows what was launched and when.
- Reused answer
- Saved answer
- Saved decision
find_task → similar: 2 failed runs for "ranker sweep"
Step by step
- ArcRoom records each training command with its flags, so the playbook knows what was launched and when.
- When a run fails, the agent records it with remember_failure: the settings and why (diverged, out of memory).
- Before proposing new settings, the agent checks find_task and gets the failed ones back first.
- Settings that worked are saved with complete_task, with the eval number the agent read.
What the agent sees
find_task → similar: 2 failed runs for "ranker sweep"
• lr 1e-3: loss NaN at step ~4k (Oct 9)
• bs 512: CUDA out of memory on 1×A10 (Oct 11)
Tried and worked: lr 3e-4, bs 256, ndcg@10 0.412Compared with today's tools
Experiment tracker (MLflow or Weights & Biases)
Every run logs params and metrics to the tracker. Before a new sweep, someone opens the runs table and tells the agent what to avoid.
- Setup
- 1–2 hours
- Ongoing
- look up past runs and paste them in, ~10 min per sweep
ArcRoom
Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.
- Setup
- 2 min (npx arcroom init)
- Ongoing
- None
The tracker is the better home for metrics and curves, but the agent doesn't read it on its own; ArcRoom puts the failed settings in front of the agent before it proposes the same run.
Where it stops
ArcRoom sees the command, not the training logs or loss curves. Only results the agent saves with remember_failure or complete_task are known.
Try it in your project
Works with the agents your team already uses, and any agent over MCP or the HTTP API.