How I handed most of the E2E test work to AI agents, and what still needs a person

01 · The problem

Every failure asks one question, and answering it takes a lot of reading.

I lead a team that investigates 5,000+ E2E test failures a day for about 200 customers.6 Each one asks the same question: is the E2E test wrong, or is the product broken? Answering it means reading the logs, the trace and the live page, finding the cause with evidence, and proving the fix.

New E2E tests were slow too: the company expected about 10 a week.3 And every E2E test, new or a teammate's, needs someone to check that it does what its steps say.

I wanted agents to do that reading and leave the decision to a person, so I could lead the team and still investigate hands-on all day.7

Before this project I had already made AI output testable: 65 E2E tests that check non-deterministic LLM behavior, even though the answers change every run.8 Mission Control asks the next question: can agents do the investigating, the creating and the reviewing, and stay honest about it?

02 · The constraints

What the system had to respect

• A person approves what is hard to undo. Changes to a customer's E2E tests, and anything sent to a customer, wait for me.
• An agent saying "fixed" is not proof. Only a run shows that a fix works.9
• Sessions end. A session can close mid-task, and its work must not vanish with it.4
• Context costs money. A raw snapshot of a live page costs 3-8k tokens.5
• I still lead a team. The system runs while I work, and shows me only what needs me.

03 · The decisions

The decisions, and what each one cost

Each choice traded something away.

1. Re-run before reading

   Every new batch of failures starts with a plain re-run, before any agent reads a trace.

   Cost

   The first diagnosis waits for the re-run.

   In return

   A failure that passes with no change costs no investigation.

2. Proof before "ready"

   "Ready" needs a clean local run plus a run on the platform's own infrastructure, on the same code, with the run id attached.9

   Cost

   Every fix waits for both runs.

   In return

   "Ready" can be checked, not just trusted.

3. I apply the fixes myself

   Agents find the cause and prove the fix. I apply the change to a customer's E2E tests.2

   Cost

   A proven fix can wait for me.

   In return

   No customer's E2E test changes without a person.

4. A lead per client, and everything in plain text

   One lead agent session owns each client, worker sessions each own one piece of work end to end, and all state lives in plain-text records.4

   Cost

   No agent keeps its progress in its head. It writes everything down as it goes.

   In return

   No work is lost when a session ends, and clients run side by side: 20 at once, about 800 failures in all.4

5. Show the agent less

   A small tool reads the live page and hands the agent a filtered report, not a raw snapshot.

   Cost

   One more tool to build and maintain.

   In return

   About 1k tokens per report instead of 3-8k, and a live page inspected in 17s instead of 62s.5

6. Guardrails in the tool, not in the prompt

   The tool itself refuses destructive actions on the test platform, whatever an agent was told.9

   Cost

   A refused command stops its agent until I allow it, so refusals show in red on the page I watch.

   In return

   Safety does not depend on a model following instructions.

7. Every correction becomes a rule, and a check where it can

   When I correct an agent, the lesson goes into its rules and, where it can, becomes a check that fails on a repeat.

   Cost

   Checks pile up and slow every change. So a check runs only where a change reaches, and the full list runs once, when a change touches many files.

   In return

   A corrected mistake is caught by a check next time, not by me.

04 · How it runs

Three kinds of work, leads and workers, two pages

The agents run on Claude Code and work the test platform through its CLI and MCP.2 A lead agent session owns each client, and workers take one piece each, from writing a fix to proving it.4

• Investigate. Each client's lead re-runs its failures, groups them by cause and finds each root cause with evidence.4
• Create. Creation agents write new E2E tests from their written steps, and every step must map to an assertion seen passing.3
• Review. Review agents check an E2E test against its written steps, a new one or a teammate's, and a second agent challenges every verdict.3

The Bridge is the one page I watch: what needs a person, what each session is doing, what is waiting and what got done.

Investigations: hundreds of failures at once, grouped by cause, worked in parallel, each fix proven before it reaches me. A product change or bug question comes to me there, and a bug report takes one key, not 20 to 30 minutes.2

05 · How it stays honest

Nothing counts until something can check it

Two of the decisions above already keep it honest: proof before "ready", and guardrails in the tool.9 Three more things help.

• Every created E2E test proves its steps. Each written step must map to an assertion seen passing, and a second agent challenges every review verdict.3
• Everything is on the record. Every request, reply and model call lands in an inbox with a derived status and a per-day cost ledger.5
• A new check must fail first. Before a check is trusted, it has to catch a planted mistake.

06 · One day

One day, start to finish

An invented day: Client A, B and C, and made-up E2E test ids. The steps are the real ones.

1. MorningA suite run leaves failures for three clients. Each client's lead re-runs them first; those that pass unchanged drop out.
2. Mid-morningThe rest are grouped by cause. One of Client A's groups traces to a renamed checkout field. A worker writes the fix and proves it locally and on the platform, and it reaches The Bridge as ready.
3. Late morningClient B's srch-02 fails because search now sorts by date, while its steps expect relevance. Product change or bug is my call, so Investigations asks me.
4. AfternoonThe bug report for exp-02, an empty export, is drafted from the evidence. A worker creates new E2E tests for Client C, and a review agent challenges each verdict.
5. Late afternoonA teammate's E2E test for Client A comes in for review. An agent checks each written step against the code, a second agent challenges the verdict, and the note to my teammate waits for me.
6. End of dayI apply the proven fixes and run them on the platform myself; my run is the final word. Anything open waits in its record for tomorrow.

07 · What still needs a person

The calls that stay with me

The agents do the reading, the writing and the first review. These stay with me:

• Applying a change to a customer's E2E tests.
• Whether a changed behavior is a product change or a bug.
• Anything that leaves the building, like a bug report or a message to a customer.
• The note on a teammate's E2E test: I read it and send it.
• The final run's verdict: a pass is mine to mark.
• What to work on next.

08 · What changed

The reading and the writing moved to the agents. The calls stayed with a person.

300-500

failing E2E tests cleared a day on one large AI product, in about four hours, while I lead a team7

50

fully vetted E2E tests in one day, against 10 a week expected3

1 key

for a bug report, not 20 to 30 minutes2

17s

to inspect a live page, down from 62s5

Sources

Every figure on this page is on my resume.

Each footnote lands here. Each line names where its figure sits on the resume and opens that line.

1. 1A personal project, 2026 to presentResume · Projects, Mission Control · its datesSee it on the resume
2. 210x as efficient as by hand, hundreds of failures at once through the platform's CLI and MCP, each fix verified before I apply it, and a bug report in one key, not 20 to 30 minutesResume · Projects, Mission Control · bullet 1See it on the resume
3. 350 fully vetted E2E tests in one day, against 10 a week expected, with creation and review agents: every written step mapped to an assertion seen passing, and a second agent challenging every review verdictResume · Projects, Mission Control · bullet 2See it on the resume
4. 420 client investigations at once, about 800 failures in all, with a lead session per client, workers and plain-text recordsResume · Projects, Mission Control · bullet 3See it on the resume
5. 5~1k tokens per filtered report, down from 3-8k, live-page inspection from 62s to 17s, and the inbox and cost ledgerResume · Projects, Mission Control · bullet 5See it on the resume
6. 65,000+ E2E test failures a day, for about 200 customersResume · Lead Software QA Engineer · bullet 1See it on the resume
7. 7300-500 failing E2E tests cleared a day on one large AI product, in about four hours, while leading the team and still investigating hands-onResume · Lead Software QA Engineer · bullet 2See it on the resume
8. 865 E2E tests that check non-deterministic LLM behaviorResume · Software QA Engineer · bullet 2See it on the resume
9. 9No agent certifies its own fix: "ready" needs a clean local run plus a platform run, same code, run id attached, and guardrails refuse destructive platform actions at the tool boundaryResume · Projects, Mission Control · bullet 4See it on the resume