A prompt that gets better each week

Building agentsBuilder tuning the system prompt of a support-reply agent

I tune our reply agent's prompt with Claude Code: change a line, run 30 sample tickets, compare. After three weeks I had 40 versions, and I kept retrying wording I had already ruled out.

How do I keep track of which system prompt versions I already tried and ruled out?

Save each prompt version and its score with ArcRoom's complete_task, and wording that made things worse with remember_failure and why. Record the final choice with arcroom decide and a --why reason, so later sessions don't undo it. Next week, recall brings the kept version and the ruled-out ones before you start. ArcRoom doesn't run tests or score prompts; you still need your own eval.

Also asked as: I keep retrying prompt wording I already ruled out. How do I stop? · How do I keep a log of prompt changes and results while tuning an agent? · Can Claude Code remember which prompt versions did worse?

The work

Each prompt version and its score is saved with complete_task.

ArcRoom keeps
  • Saved answer
  • Decision
  • Saved decision
  • Recall
The next agent gets

$ arcroom decide "keep tone line v14: plain, no apologies unless at fault" --why "+9% on 30 tickets; v15 'warmer' read as fake"

Step by step

  1. Each prompt version and its score is saved with complete_task.
  2. Wording that made things worse is saved with remember_failure and why.
  3. The final choice is recorded with arcroom decide "…" --why …, so later sessions don't undo it.
  4. Next week, recall brings the kept version and the ruled-out ones before I start.

What the agent sees

$ arcroom decide "keep tone line v14: plain, no apologies unless at fault" --why "+9% on 30 tickets; v15 'warmer' read as fake"
Next session, recall: v14 kept · v15, v17 ruled out

Compared with today's tools

A prompt version spreadsheet

Keep every version, score and note in a sheet and paste the best one into each session.

Setup
30 min
Ongoing
Fill the sheet after each try, ~5 min

ArcRoom

Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.

Setup
2 min (npx arcroom init)
Ongoing
None

A sheet is easy to share and sort; the agent doesn't read it on its own, so ruled-out wording comes back.

Where it stops

ArcRoom records what was tried and decided; it doesn't run the tests or score the prompts. You still need your own eval.

Try it in your project

Works with the agents your team already uses, and any agent over MCP or the HTTP API.

More in building agents