Kubernetes debugging, the second time
DevOps, SRE and securitySRE debugging a cluster with an agent in the terminal
After a node pool upgrade, our worker pods went into CrashLoopBackOff. It took two hours to find the cause: the new nodes had a smaller ephemeral-storage limit. Four months later the next upgrade did the same, and the agent started by raising memory limits, which hadn't helped the first time either.
How do I keep an AI agent from retrying fixes that failed last time we debugged Kubernetes?
Save approaches that didn't help with ArcRoom's remember_failure and the reason, and the cause and fix with complete_task once it's solved. Next time, find_task returns both before the agent starts guessing. ArcRoom records the kubectl and helm commands it ran, not their output. The agent still reads live pod events and logs.
Also asked as: The same CrashLoopBackOff came back and the agent tried the old wrong fix first. How do I stop that? · How do I save what we learned from a Kubernetes incident for the next one? · Can an AI agent remember how we fixed a cluster issue months ago?
ArcRoom records the kubectl and helm commands the agent ran, not their output.
- Reused answer
- Saved answer
- Saved decision
find_task → similar: "worker CrashLoopBackOff after node upgrade" (Jun 18)
Step by step
- ArcRoom records the kubectl and helm commands the agent ran, not their output.
- Approaches that didn't help are saved with remember_failure and the reason.
- The cause and fix are saved with complete_task once it's solved.
- Next time, find_task returns both before the agent starts guessing.
What the agent sees
find_task → similar: "worker CrashLoopBackOff after node upgrade" (Jun 18)
Failed: raise memory limit to 2Gi (no change)
Fix: ephemeral-storage request 4Gi, node pool disk 100GiCompared with today's tools
Slack thread + team wiki
The fix is posted in the incident channel and, if someone has time, written up on the wiki.
- Setup
- None
- Ongoing
- search Slack and paste into the agent, ~10 min; often not found
ArcRoom
Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.
- Setup
- 2 min (npx arcroom init)
- Ongoing
- None
Slack keeps the fix but not the dead ends, and the agent never searches it; ArcRoom returns both inside the agent's session.
Where it stops
The agent still reads live pod events and logs. ArcRoom stores only what was saved, never kubectl output.
Try it in your project
Works with the agents your team already uses, and any agent over MCP or the HTTP API.