By David Turner
Imagine an AI coding agent changes a workflow’s readiness logic, generates tests around the change, reviews its own diff, and reports that the feature is ready to ship. Every step may look reasonable. The implementation and the tests may even agree.
That agreement is not necessarily independent evidence.
The dangerous case is not simply that an AI system “hallucinates.” It is that generation and validation can share the same assumptions, context, and blind spots. When that happens, a mistake can become self-consistent: the implementation encodes it, the generated tests expect it, and the evaluator rewards the resulting agreement.
Recent preprints make this concern unusually concrete. In On the Risk of Coding Before Testing, researchers report error propagation when LLMs generate code before generating its tests: faults in the implementation can be reproduced in the test artifacts, allowing incorrect code and incorrect tests to agree with each other. A separate 2026 study describes a related “misguidance effect,” in which buggy code steers LLM-generated unit tests toward validating erroneous behavior rather than exposing it.
The lesson is not “AI cannot check AI.” Model-based evaluation can be extremely useful. Work on LLM-as-a-judge has shown that strong models can approximate human preferences surprisingly well, while also documenting position, verbosity, and self-enhancement biases. Other research has found that LLM evaluators can recognize and favor their own generations. The useful distinction is therefore not AI versus human. It is assessment versus evidence.
The real question is failure-mode independence
Software testing has always had an oracle problem: observing a program’s behavior is not enough unless we have some basis for deciding whether that behavior is correct. A test can execute perfectly while asserting the wrong thing. A deterministic validator can consistently enforce the wrong requirement.
That is why “deterministic” should not be confused with “correct.” Determinism matters because it makes a claim reproducible and falsifiable. Correctness still depends on the quality of the specification, the oracle, and the evidence collected against it.
Software assurance has long treated independence as valuable for exactly this reason. NASA’s Independent Verification and Validation program defines independence across technical, managerial, and financial dimensions, with technical independence requiring assessment separate from the developer. NIST’s AI Risk Management Framework similarly notes that independent review can improve testing and help mitigate internal bias and conflicts of interest.
For AI-assisted software, the key idea can be stated more simply:
Ask what in the validation path is capable of disagreeing with the generator if the generator’s central assumption is wrong.
If the answer is “nothing,” the process may produce confidence, but it has weak evidence independence.
An Evidence Independence Stack
Teams do not need formal verification for every AI-assisted change. They do need a validation path whose failure modes are not identical to the generator’s. A practical stack has four layers.
1. Define an acceptance contract before generation.
Write observable conditions first: valid state transitions, forbidden states, schema constraints, permission boundaries, calculation rules, accessibility requirements, or expected failure behavior. This reduces the temptation to derive the definition of “correct” from whatever implementation the model happened to produce.
2. Turn machine-verifiable requirements into executable gates.
Use type checking, static analysis, unit and integration tests, invariants, schema validation, property-based tests, or other deterministic checks where appropriate. The important point is not that these tools are infallible; it is that they can fail independently of a model’s prose judgment.
3. Add behavioral evidence with different assumptions.
Test boundaries, adversarial inputs, negative cases, and failure paths. When exact expected outputs are difficult to specify, techniques such as metamorphic testing can check necessary relationships across multiple executions rather than relying on a single expected answer. Different data, tools, implementations, and test-generation methods can all increase failure-mode diversity.
4. Put humans at the irreducible judgment boundary.
A person is most valuable where correctness cannot be reduced to a reliable machine check: product intent, ambiguity, usability, ethical tradeoffs, contextual risk, or domain-specific acceptability. But a human merely being present is not a control. The reviewer needs evidence, authority to reject the output, and clear criteria for doing so.
The stack is not a ladder where every change must reach the top. It is a way to avoid confusing repeated agreement with independent corroboration.
“Human in the loop” must be mechanically real
I encountered a small but instructive version of this problem while working on WeaveStudio, a local-first workflow tool. A guided demo could report 100% completeness / Ready while a required Review node still showed Pending. The interface contained a human-review concept, but the state machine did not actually make that review a readiness gate.
The fix was not a more persuasive warning. It was mechanical: a pending or rejected required review cannot produce Ready; semantic changes invalidate prior approval; generation re-checks readiness; and regression tests protect those rules. The public issue documents both the defect and the fix.
That distinction matters. Human presence means a person appears somewhere in the process. Human control means the system cannot cross a consequential boundary without the required decision. Human validation means that decision is informed by evidence and appropriate expertise.
A checkbox labeled “Reviewed” is not assurance if the software can ignore it.
Validation rigor should scale with consequence
Not every AI-generated change deserves the same assurance budget. A CSS spacing adjustment and an authorization rule should not pass through identical gates.
Four factors should increase validation intensity: uncertainty, consequence, irreversibility, and poor observability. A change that is easy to roll back, immediately visible, and low-impact may need only lightweight checks. Authentication logic, financial calculations, destructive migrations, regulated workflows, or security controls warrant stronger and more independent evidence.
This is where ordinary engineering controls become especially valuable. GitHub, for example, can require specified status checks to pass before a protected branch is merged. The principle is larger than any platform: if a condition is important and machine-verifiable, make it an enforced gate rather than a recommendation in a prompt.
A practical pattern for tomorrow
For teams using AI coding assistants or agents, the workflow can be compact:
- Write observable acceptance criteria before asking for implementation.
- Let the model generate candidates, not the definition of success after the fact.
- Convert machine-verifiable criteria into executable checks.
- Use model critique for semantic review, alternatives, and anomaly spotting – not as the sole proof that the change is correct.
- Add test methods, data, or reviewers with genuinely different failure modes when consequence rises.
- Preserve the evidence: which commit, specification, tests, results, and human decisions justified release.
- Revalidate when semantic inputs change; do not let stale approval survive a materially different artifact.
AI-assisted development can move extraordinarily fast. The assurance process has to move with it, but speed does not require self-certification.
The question to ask after an AI system says “this is ready” is not whether the explanation sounds confident. It is whether the evidence path contains something capable of proving the generator wrong.
Confidence is an output. Evidence is a property of the validation process.
Author Bio
David Turner is an independent applied-AI and workflow builder focused on local-first software, validation, provenance, and technical operations. His recent projects explore how AI assistance can be bounded by explicit review states, deterministic checks, and traceable evidence. He publishes his work and technical case studies on GitHub.
