What I learned running Claude inside a real QA pipeline

A lab demo and a production pipeline disagree about what “the model did well” means.

In a demo, the model writes a Playwright spec for a toy app, the spec passes, and someone takes a screenshot of the code. In a pipeline, the model is one more worker with incomplete context, a flaky environment, a corpus it is not allowed to see, and a human who will be asked — later, in an incident — why this check exists.

I have been running Claude inside a real ebook QA pipeline. Not as a mascot. As staff. These are the notes I wish I had on week one.

Give it a job, not a profession

The first prompt we should not have written was “act as a senior QA engineer.” That prompt produces a confident generalist who wants to redesign your suite.

The prompts that work look like job tickets. Check this class of structural defect. Return JSON in this schema. If you are unsure, say so in this field; do not invent a locator. Do not propose deletions. Do not “fix” an assertion to match the file. Stop.

This is management, not prompt craft. A contractor who can do anything will do the wrong thing at 2 a.m. A contractor with a ticket will still be wrong sometimes, but you can see where.

Claude is better than I expected at describing a failure. It is worse than I expected at knowing when it does not have enough context. The job ticket has to make silence a valid output. Otherwise you get locators from a neighboring page and assertions that sound like English.

Selectors rot. Assertions are the meaning.

If you let the model spend its attention anywhere, it will spend it on selectors. Selectors are visible. They fail loudly. They are also the cheapest part of a test to repair, and the least related to whether the product still keeps its promise.

The interesting work is the assertion. What would have to be true for this ebook to be shippable. What is an oracle versus a heuristic. What is allowed to change between builds.

I started reviewing model output in that order. Assertion first. Selector second. Prose never. If the assertion is vague — “the page looks correct,” “the summary is good” — the test is not a test. It is a comment. The model will write a lot of comments if you let it, and they will pass.

A useful surprise: Claude is often stronger at proposing an assertion from a requirement than it is at finding a stable locator in a messy DOM. That matches the shape of the training. It has read more about meaning than it has suffered your front end. Use that. Do not ask it to be your Selenium IDE.

The evaluator is software. Test it.

Once a model grades other output — summaries, extracted metadata, “is this alt text adequate” — you have a second system under test. Teams forget.

We caught this the usual way. A silent prompt change. A batch that looked slightly kinder. A class of defects that stopped being flagged, not because the books improved. Because the grader had drifted.

So the grader has goldens. The grader has a regression set. The grader has a version. Changing the grader is a change to the quality system, and it goes through the same signature as a change to the suite. This feels heavy until the first time you can answer “what changed” in a sentence.

If you are using the Claude API as a judge, log the model id, the prompt hash, and the threshold. Future you is an auditor. Future you is tired.

Humans sign. Models propose.

I will not ship a release on a model’s say-so. That is not a ethics speech. It is an operations speech. When a storefront has a bad file, nobody wants to hear that the agent was pretty sure.

The human signature is not a rubber stamp if the harness is built to show a diff, a sample, and a reason. It is a rubber stamp if the UI is a green check. Design the review surface so that a tired person can disagree.

This also decides how you staff. You need fewer people writing the same check for the thousandth time. You need more people who can look at a model’s proposal and say “that oracle is wrong.” That second skill is the career. I will write about the org chart separately.

What I would not do again

I would not start with coverage. I would start with a list of defects that actually shipped, and I would refuse to expand the agent’s job until it could speak to those.

I would not let the model edit tests in place on day one. Proposals in a branch. Always. Deletion is a privilege you earn after you have proven you can add.

I would not mix deterministic checks and model checks in a way that hides which is which. Format validity, links, navigation: code. Language, judgment, “does this read like the print edition”: model, with a rubric. If you blur them, you cannot tell why you failed, and you cannot tell an auditor what the system is.

The pipeline is less magical than the demo. It is also the first version of this work I would defend in a room that has to live with the output. That is the only bar I have found that matters.