Flaky tests, known as flaky
CI and automationTeam with a slow, noisy test suite
Every CI agent that hit test_export_timeout spent 20 minutes 'fixing' a test that's simply flaky.
How do I stop CI agents from spending ages trying to fix a test that is just flaky?
Have the first agent that finds a flaky test save it with ArcRoom's remember, with a note like 'rerun once; don't change code'. Later agents get that memory when they hit the test, and rerun it once instead of rewriting it. The playbook only stops the wasted effort; someone should still fix the flaky test.
Also asked as: Our agents keep spending 20 minutes 'fixing' a flaky test. How do we stop it? · How do I tell AI agents which tests are known to be flaky? · Can one agent warn the next one about a flaky test?
The first agent that finds a test flaky saves it with remember.
- Saved decision
Remember: test_export_timeout is flaky on CI (shared runner, 1 in 8). Rerun once; don't change code.
Step by step
- The first agent that finds a test flaky saves it with remember.
- Later agents get it when they hit that test.
- They rerun it once instead of rewriting it.
What the agent sees
Remember: test_export_timeout is flaky on CI (shared runner, 1 in 8). Rerun once; don't change code.Compared with today's tools
Retries or quarantine in the test runner (jest.retryTimes, pytest-rerunfailures)
The team marks the test as flaky so the runner retries it, and the build stops going red for it.
- Setup
- 15 min
- Ongoing
- Tag each flaky test once
ArcRoom
Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.
- Setup
- 2 min (npx arcroom init)
- Ongoing
- None
A runner retry helps every agent and person, and is the better fix. ArcRoom helps when nobody has tagged the test yet, by passing on the first agent's finding.
Where it stops
Someone should fix the flaky test eventually. The playbook only stops the wasted effort.
Try it in your project
Works with the agents your team already uses, and any agent over MCP or the HTTP API.