Quality engineering is becoming the discipline that decides whether AI can be trusted.
For twenty years the job has been to ask a system to prove it still does what we said it would. We wrote the spec. We wrote the tests. The tests failed when the system drifted.
Agents change the shape of that contract. The system under test can now write code, change code, and — if we are not careful — write the tests that bless the change. Drift is no longer a regression in a function. It is a change in behavior that no one explicitly requested and no one can fully enumerate.
The old stack still matters. Locators still break. Data still lies. Environments still lie more. What is new is the oracle problem at scale. When a language model summarizes a thousand-page book, extracts a clause from a policy, or decides which test to run next, the question is not whether a button turned green. The question is what we would accept as evidence that this was right, and who is willing to sign it.
That is quality engineering. It always was. The tools got more interesting. The job got more serious.
Where software quality is heading
The next decade of quality will not be about writing more tests. It will be about deciding what a machine is allowed to assert.
Three shifts are already visible in production, not in keynotes.
The suite becomes a living system. Agents can generate and maintain regression. That is useful, and it is dangerous in the same motion. A suite that can rewrite itself is a suite that can launder a change. Someone still has to own the invariant.
The oracle gets harder. Deterministic output was a luxury we did not notice we had. Models return language, rankings, summaries, plans. “Equals” stops working. You need goldens, rubrics, adversarial cases, and a threshold that a human set on purpose. You need to regression-test the evaluator, because the evaluator is now software too.
Quality becomes the trust layer. Security will own a piece of this. Safety research will own a piece. Product will own a piece. The discipline that already knows how to produce evidence — for a release, for an auditor, for a post-incident review — is quality engineering. If we do not take that seat, someone less practiced at evidence will.
I do not think this makes testers obsolete. I think it makes unowned testing obsolete. The people who can write a contract, design an eval, and sign a release get more expensive. The people who only click through a script get a model as a colleague, and then as a replacement.
Why most AI-in-QA pilots fail
I have watched the same pilot fail in different companies, with different vendors. The post-mortem is almost always the same.
The pilot is scored on generation. How many tests did the model write. How fast. How much coverage went up on a dashboard. Nobody asks how many of those tests would have caught the last production incident. Nobody asks what happens when the UI changes and the model confidently updates the wrong assertion.
Pilots also fail on access. The model cannot see the staging data. It cannot see the flaky suite. It cannot see the ticket that explains why this one test is allowed to fail on Thursdays. So it writes a beautiful suite for a system that does not exist, and the team files the demo under “promising.”
The third failure is quieter. Teams skip the contract. They never write down what the agent is allowed to change, what it must escalate, and what evidence it must leave behind. In a regulated environment that omission is not a process gap. It is the reason the pilot never leaves the sandbox.
Generation is not useless. Generation without an oracle, an audit trail, and a human signer is theater. Theater is expensive. It also trains leadership to believe the work is solved.
A practitioner detail, because this is where the slide decks stop. When you put Claude or a sibling model on a Playwright suite, the model is often better at writing the assertion than the selector. Selectors rot. Assertions are the meaning. Most pilots obsess over the locators because locators are visible. The silent failure is an assertion that still passes after the product has changed what it promised.
Another: flake is a data problem disguised as a tool problem. If your suite is 12% flaky, an agent that “fixes” tests will fit to the flake. You will get a greener pipeline and a worse product. I have seen this twice. I will see it again. The fix is not a smarter model. The fix is to treat flake as a defect before you hand the suite to staff that never get tired and never get suspicious.
What I am doing about it
At a major publisher I am putting models inside a real pipeline, not a slide. Ebook quality is a useful proving ground. The artifact is linguistic and structural. The cost of a miss is public. The release cannot be hotfixed after it ships to a storefront.
The work is unglamorous. We constrain the model. We evaluate the evaluator. We keep a human on the signature. We measure whether the system finds defects we would have shipped — not whether it can emit Playwright.
The numbers that matter so far are not vanity coverage. Manual effort down about 70% on the paths the harness owns. About $200K in cost avoided against the vendor-shaped alternative. Those are real, and they are not the point. The point is a working prejudice: the model is staff. Staff need a job description, a manager, and a paper trail.
Nexacore exists because I do not think this operating model should live in one company. Every software team will need an AI quality engineer that does not sleep. That engineer will still report to a person who can explain the decision to an auditor.
I have spent twenty years in rooms that already know how to answer an auditor — insurance, healthcare, financial services, now publishing. That is the training. The new work is to bring that posture to systems that speak.
The rest of this site is evidence: the work, the bets, the notes. If you want the résumé, it is there. It is not the argument.