Fine-tuning a small model without repeating bad runs
Data, ML and researchSmall team fine-tuning a small local model
A two-person team fine-tunes a small open model on one GPU, with Claude Code and Codex driving the scripts. Over a month they learned that one LoRA setting overfits their dialogue set and that one dataset version had many duplicate lines. Each new session, in either agent, proposed the same bad setting on the same dataset again.
How do I stop Claude Code and Codex from retrying a LoRA setting that already overfit?
Save each eval result with ArcRoom's complete_task, or remember_failure with the reason, and save settled choices with arcroom decide, like 'train on v4 only'. New sessions in either agent then recall the failed setting and the decision. claim_task stops a Codex and a Claude Code session from launching the same run on your one GPU. Scores are only as good as what the agent writes.
Also asked as: Every new agent session suggests the same bad fine-tuning setup. How do we stop it? · How do I share fine-tuning results between Claude Code and Codex? · Can two AI agents avoid starting the same training run on one GPU?
ArcRoom records training and eval commands, whichever agent ran them.
- Claim
- Saved answer
- Decision
- Saved decision
Recall:
Step by step
- ArcRoom records training and eval commands, whichever agent ran them.
- After each eval on the bench, the agent saves the result with complete_task, or remember_failure with the reason.
- Settled choices are saved with arcroom decide, e.g. 'train on v4 only'.
- claim_task stops a Codex session and a Claude Code session from launching the same run on the one GPU.
What the agent sees
Recall:
• Decision: train on dataset v4 only. Why: v3 has duplicate lines
• Failed: LoRA r=64, overfits after epoch 2
• Answer: best so far r=16, 3 epochsCompared with today's tools
Training notes in a README + CLAUDE.md
Keep a hand-written log of runs and rules in the repo and point CLAUDE.md and AGENTS.md at it.
- Setup
- 30 min
- Ongoing
- update the log after every run, ~5 min; easy to forget at 2 a.m.
ArcRoom
Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.
- Setup
- 2 min per agent (npx arcroom init --agent claude|codex)
- Ongoing
- None
A hand log goes stale when nobody updates it, and Claude Code and Codex each read their own file; ArcRoom records the commands automatically and gives both agents the same memory.
Where it stops
ArcRoom never sees the weights, logs or eval outputs. Scores are only as good as what the agent writes into complete_task.
Try it in your project
Works with the agents your team already uses, and any agent over MCP or the HTTP API.