Babysitting data pipelines

Internal agentsData team with an agent watching nightly jobs

When the nightly ETL failed, an agent run from Claude Code started investigating from zero, though the same upstream schema change had broken it last month.

How can an AI agent check past failures before debugging our nightly ETL job?

Have the agent check ArcRoom's find_task for earlier failures of the job, so known causes are checked first, like an upstream column change fixed by updating the schema map. The new cause and fix are saved for next time. This works today when the agent runs in Claude Code or Codex. A custom agent needs the SDK.

Also asked as: Our ETL broke the same way as last month and the agent started from zero. Why? · How do I give a pipeline-watching agent memory of past failures? · Can an AI agent remember how a failed data job was fixed before?

The work

The agent checks find_task for earlier failures of this job.

ArcRoom keeps
  • Reused answer
The next agent gets

find_task → similar: "etl_orders failed" (Sep 30): upstream added column; fix: update schema map

Step by step

  1. The agent checks find_task for earlier failures of this job.
  2. Known causes are checked first.
  3. The new cause and fix are saved.

What the agent sees

find_task → similar: "etl_orders failed" (Sep 30): upstream added column; fix: update schema map

Compared with today's tools

A runbook in the repo

The team keeps RUNBOOK.md with known failure causes per job and tells the agent to read it first.

Setup
1 hour
Ongoing
Update the runbook after each incident, ~10 min

ArcRoom

Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.

Setup
2 min (npx arcroom init)
Ongoing
None

A runbook is reviewed and people use it too. It helps only if someone writes it up, while ArcRoom saves the cause and fix when the agent finishes the task.

Where it stops

Works today if the agent is Claude Code or Codex. A custom agent needs the SDK.

Try it in your project

Works with the agents your team already uses, and any agent over MCP or the HTTP API.

More in internal agents