Fine-tuning a small model without repeating bad runs

Data, ML and researchSmall team fine-tuning a small local model

A two-person team fine-tunes a small open model on one GPU, with Claude Code and Codex driving the scripts. Over a month they learned that one LoRA setting overfits their dialogue set and that one dataset version had many duplicate lines. Each new session, in either agent, proposed the same bad setting on the same dataset again.

How do I stop Claude Code and Codex from retrying a LoRA setting that already overfit?

Save each eval result with ArcRoom's complete_task, or remember_failure with the reason, and save settled choices with arcroom decide, like 'train on v4 only'. New sessions in either agent then recall the failed setting and the decision. claim_task stops a Codex and a Claude Code session from launching the same run on your one GPU. Scores are only as good as what the agent writes.

Also asked as: Every new agent session suggests the same bad fine-tuning setup. How do we stop it? · How do I share fine-tuning results between Claude Code and Codex? · Can two AI agents avoid starting the same training run on one GPU?

The work

ArcRoom records training and eval commands, whichever agent ran them.

ArcRoom keeps
  • Claim
  • Saved answer
  • Decision
  • Saved decision
The next agent gets

Recall:

Step by step

  1. ArcRoom records training and eval commands, whichever agent ran them.
  2. After each eval on the bench, the agent saves the result with complete_task, or remember_failure with the reason.
  3. Settled choices are saved with arcroom decide, e.g. 'train on v4 only'.
  4. claim_task stops a Codex session and a Claude Code session from launching the same run on the one GPU.

What the agent sees

Recall:
• Decision: train on dataset v4 only. Why: v3 has duplicate lines
• Failed: LoRA r=64, overfits after epoch 2
• Answer: best so far r=16, 3 epochs

Compared with today's tools

Training notes in a README + CLAUDE.md

Keep a hand-written log of runs and rules in the repo and point CLAUDE.md and AGENTS.md at it.

Setup
30 min
Ongoing
update the log after every run, ~5 min; easy to forget at 2 a.m.

ArcRoom

Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.

Setup
2 min per agent (npx arcroom init --agent claude|codex)
Ongoing
None

A hand log goes stale when nobody updates it, and Claude Code and Codex each read their own file; ArcRoom records the commands automatically and gives both agents the same memory.

Where it stops

ArcRoom never sees the weights, logs or eval outputs. Scores are only as good as what the agent writes into complete_task.

Try it in your project

Works with the agents your team already uses, and any agent over MCP or the HTTP API.

More in data, ml and research