Flaky tests, known as flaky

CI and automationTeam with a slow, noisy test suite

Every CI agent that hit test_export_timeout spent 20 minutes 'fixing' a test that's simply flaky.

How do I stop CI agents from spending ages trying to fix a test that is just flaky?

Have the first agent that finds a flaky test save it with ArcRoom's remember, with a note like 'rerun once; don't change code'. Later agents get that memory when they hit the test, and rerun it once instead of rewriting it. The playbook only stops the wasted effort; someone should still fix the flaky test.

Also asked as: Our agents keep spending 20 minutes 'fixing' a flaky test. How do we stop it? · How do I tell AI agents which tests are known to be flaky? · Can one agent warn the next one about a flaky test?

The work

The first agent that finds a test flaky saves it with remember.

ArcRoom keeps
  • Saved decision
The next agent gets

Remember: test_export_timeout is flaky on CI (shared runner, 1 in 8). Rerun once; don't change code.

Step by step

  1. The first agent that finds a test flaky saves it with remember.
  2. Later agents get it when they hit that test.
  3. They rerun it once instead of rewriting it.

What the agent sees

Remember: test_export_timeout is flaky on CI (shared runner, 1 in 8). Rerun once; don't change code.

Compared with today's tools

Retries or quarantine in the test runner (jest.retryTimes, pytest-rerunfailures)

The team marks the test as flaky so the runner retries it, and the build stops going red for it.

Setup
15 min
Ongoing
Tag each flaky test once

ArcRoom

Every agent reads the team playbook before it works and writes to it when it finishes. Nothing to update by hand.

Setup
2 min (npx arcroom init)
Ongoing
None

A runner retry helps every agent and person, and is the better fix. ArcRoom helps when nobody has tagged the test yet, by passing on the first agent's finding.

Where it stops

Someone should fix the flaky test eventually. The playbook only stops the wasted effort.

Try it in your project

Works with the agents your team already uses, and any agent over MCP or the HTTP API.

More in ci and automation