Which settings already failed

Data, ML and researchML engineer running training experiments through Claude Code

We were tuning a ranking model over three weeks. Learning rate 1e-3 diverged by step 4,000, and batch size 512 ran out of GPU memory. A week later a new session suggested both again, and we lost an evening finding out twice.

How do I stop Claude Code from suggesting training settings that already failed?

Have the agent record each failed run with ArcRoom's remember_failure: the settings and why, like diverged or out of memory. Before proposing new settings, it checks find_task and gets the failed ones back first, along with settings that worked, saved with complete_task. ArcRoom records each training command and its flags, not logs or loss curves, so only saved results are known.

Also asked as: My agent keeps proposing a learning rate that already diverged. How do I fix that? · How do I keep a record of failed hyperparameter runs that AI agents can read? · Can Claude Code remember which experiment settings ran out of GPU memory?

The work

ArcRoom records each training command with its flags, so the playbook knows what was launched and when.

ArcRoom keeps
  • Reused answer
  • Saved answer
  • Saved decision
The next agent gets

find_task → similar: 2 failed runs for "ranker sweep"

Step by step

  1. ArcRoom records each training command with its flags, so the playbook knows what was launched and when.
  2. When a run fails, the agent records it with remember_failure: the settings and why (diverged, out of memory).
  3. Before proposing new settings, the agent checks find_task and gets the failed ones back first.
  4. Settings that worked are saved with complete_task, with the eval number the agent read.

What the agent sees

find_task → similar: 2 failed runs for "ranker sweep"
• lr 1e-3: loss NaN at step ~4k (Oct 9)
• bs 512: CUDA out of memory on 1×A10 (Oct 11)
Tried and worked: lr 3e-4, bs 256, ndcg@10 0.412

Compared with today's tools

Experiment tracker (MLflow or Weights & Biases)

Every run logs params and metrics to the tracker. Before a new sweep, someone opens the runs table and tells the agent what to avoid.

Setup
1–2 hours
Ongoing
look up past runs and paste them in, ~10 min per sweep

ArcRoom

Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.

Setup
2 min (npx arcroom init)
Ongoing
None

The tracker is the better home for metrics and curves, but the agent doesn't read it on its own; ArcRoom puts the failed settings in front of the agent before it proposes the same run.

Where it stops

ArcRoom sees the command, not the training logs or loss curves. Only results the agent saves with remember_failure or complete_task are known.

Try it in your project

Works with the agents your team already uses, and any agent over MCP or the HTTP API.

More in data, ml and research