Babysitting data pipelines
Internal agentsData team with an agent watching nightly jobs
When the nightly ETL failed, an agent run from Claude Code started investigating from zero, though the same upstream schema change had broken it last month.
How can an AI agent check past failures before debugging our nightly ETL job?
Have the agent check ArcRoom's find_task for earlier failures of the job, so known causes are checked first, like an upstream column change fixed by updating the schema map. The new cause and fix are saved for next time. This works today when the agent runs in Claude Code or Codex. A custom agent needs the SDK.
Also asked as: Our ETL broke the same way as last month and the agent started from zero. Why? · How do I give a pipeline-watching agent memory of past failures? · Can an AI agent remember how a failed data job was fixed before?
The agent checks find_task for earlier failures of this job.
- Reused answer
find_task → similar: "etl_orders failed" (Sep 30): upstream added column; fix: update schema map
Step by step
- The agent checks find_task for earlier failures of this job.
- Known causes are checked first.
- The new cause and fix are saved.
What the agent sees
find_task → similar: "etl_orders failed" (Sep 30): upstream added column; fix: update schema mapCompared with today's tools
A runbook in the repo
The team keeps RUNBOOK.md with known failure causes per job and tells the agent to read it first.
- Setup
- 1 hour
- Ongoing
- Update the runbook after each incident, ~10 min
ArcRoom
Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.
- Setup
- 2 min (npx arcroom init)
- Ongoing
- None
A runbook is reviewed and people use it too. It helps only if someone writes it up, while ArcRoom saves the cause and fix when the agent finishes the task.
Where it stops
Works today if the agent is Claude Code or Codex. A custom agent needs the SDK.
Try it in your project
Works with the agents your team already uses, and any agent over MCP or the HTTP API.