An eval loop that reads its own past failures
Building agentsTeam improving an agent against a 120-task eval set
We run our agent, built on Claude Code, against 120 eval tasks after each change. We wanted each round of fixes to start from what failed last round: which tasks, which error, which fix we already tried.
How do I start each round of agent eval fixes from what failed and what I tried last round?
Save each failed eval task with ArcRoom's remember_failure: the task id, the error and the change being tested. Before the next round, the fixing session uses find_task to list past attempts for tasks still red, and fixes that worked are saved with complete_task. For the scored run itself, run arcroom off, then arcroom on again, so memory can't leak answers into the test.
Also asked as: We keep retrying fixes for eval tasks that already failed. How do we track them? · How do I keep a history of fix attempts for each eval task? · Can agent memory inflate my eval scores?
Each failed eval task is saved with remember_failure: the task id, the error and the change being tested.
- Reused answer
- Saved answer
- Saved decision
- Off switch
find_task → similar (2) for eval t-047 'parse invoice dates':
Step by step
- Each failed eval task is saved with remember_failure: the task id, the error and the change being tested.
- Before the next round, the fixing session uses find_task to list past attempts for the tasks still red.
- Fixes that turned a task green are saved with complete_task.
- For the scored run itself, run arcroom off, then arcroom on again, so memory can't leak answers into the test.
What the agent sees
find_task → similar (2) for eval t-047 'parse invoice dates':
✗ round 6: prompt fix only, still fails on dd/mm
✗ round 8: added examples, fails on '3rd Oct'Compared with today's tools
A results table (DIY)
Log each round's pass or fail per task to a sheet or database and compare rounds by hand.
- Setup
- Half a day
- Ongoing
- Write up what was tried, ~20 min a round
ArcRoom
Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.
- Setup
- 2 min (npx arcroom init)
- Ongoing
- None
An eval table is the right source of truth for scores; it doesn't put past attempts in front of the agent fixing the next one.
Where it stops
Memory and clean evals don't mix: a scored run with recall on can reuse answers and inflate the score. Keep the scored run separate.
Try it in your project
Works with the agents your team already uses, and any agent over MCP or the HTTP API.