Why most AI testing pilots fail

The first time I watched an AI testing pilot die, the model was fine. The vendor was fine. The executive sponsor still used the word “promising” in the readout. Six weeks later the suite the model had written was unmaintained, the original testers were back in the spreadsheet, and the license was “under review.”

I have now seen this enough times to stop being surprised.

The failure is rarely that the model cannot write a test. Writing a test is a solved-enough problem if you are willing to accept mediocre locators and a cheerful assertion. The failure is that the pilot is designed to produce a demo, and demos have the wrong oracle.

Scored on the wrong thing

Pilots get measured on volume. Tests generated per hour. Percentage of the backlog “automated.” Lines of Playwright. Coverage that went up because the tool added checks nobody asked for.

None of that is quality.

A useful score would be: of the last ten defects that reached production, how many would this system have caught, and at what cost in flake and noise. That score is embarrassing, so it is not used. It requires a corpus of real incidents, which requires a team that writes incidents down, which is already a quality practice most pilots skip.

I asked one team to take their last Sev-2 and replay it against the generated suite. The suite did not have a test for that path. The model had spent its budget on a login page that three other tools already owned. Generation without a risk model is a random walk with better grammar.

No access, no memory

The second failure is environmental, and it is boring, which is why it keeps winning.

The model cannot see staging data. It cannot see the flake report. It cannot see the Jira comment that says “this fails on the Thursday job because of a vendor file, do not touch.” It cannot see that Tosca owns this journey and Playwright owns that one, and the reason is political as much as technical.

So it writes a suite for a system that does not exist: clean DOM, stable IDs, deterministic responses, a product that never A/B tests a button. The demo environment is that system. Production is not.

A practitioner fix, not a slogan: before you let an agent write tests, give it the last 90 days of failures as context, or do not bother. If you cannot export that, you do not have a data problem. You have a quality system that is not ready to be staffed by anyone, human or otherwise.

The missing contract

The third failure is the one I care about, because it is the one that will get someone in regulated work into trouble.

Nobody writes down what the agent is allowed to do.

May it add a test? Edit a test? Delete a test that is “redundant”? Open a pull request? Merge? Change an assertion to match new behavior it inferred from the UI? That last one is how you launder a product bug into a green pipeline.

In a GxP-adjacent or SOX-adjacent environment, “the model changed the suite” is not an origin story an auditor wants. You need a policy: allowed actions, required evidence, required human signature above a threshold. That policy is the product. The model is a worker inside it.

I have started calling this a quality contract. It is not a prompt. Prompts drift. A contract is versioned, reviewed, and owned by the same function that owns the release signature.

What the pilots that survive do differently

The ones that last — there are a few — look like staffing, not like magic.

They pick a narrow job. Not “be QA.” Something like: propose checks for this class of ebook structural defect, in this format, against this oracle, and stop. They measure defects found that would have shipped. They keep a human on the merge. They treat flake as a blocker to expansion, not as noise to be fitted away.

They also budget for the unglamorous work. Eval sets. Failure taxonomies. A place to put the tribal knowledge the model cannot see. If that work is “later,” later never comes, and the pilot dies on the second product change.

I am doing this now, in production, on ebook quality. Manual effort is down on the paths the harness owns. The number I watch is not tests written. It is whether a human still understands why a check exists, and whether we can show that to someone who was not in the room.

If your AI testing pilot cannot survive that question, it is not a quality system. It is a content generator with a CI plugin. Those are easy to buy. They are also easy to turn off, which is what happens, quietly, after the readout.