Daviyon DanielsMost AI governance conversations stop at the model. Which model, trained on what, evaluated against which benchmark. That framing misses the question that actually determines whether an AI system can be trusted in production: who checks its work, and are they in a position to catch what it gets wrong?
In most deployments, the answer is nobody. The model produces an output, the output becomes a record, and the record becomes the basis for a decision. Where there is a check at all, it is frequently the same model — or a second instance of it — asked to evaluate its own reasoning.
To test that assumption properly, we built a system where it could be measured: an autonomous fleet that owns the vulnerability remediation lifecycle, with every decision it makes recorded and reviewable. The result was a number we did not expect and initially misread as a defect.
The setup: two models, deliberately different families
The system triages security findings. A reasoning agent proposes a severity, a remediation deadline and a remediation path, citing evidence from CISA's Known Exploited Vulnerabilities catalog, EPSS exploitation-probability scores and NVD severity data.
Nothing that agent proposes becomes state until a second agent reviews it. The reviewer either ratifies the proposal or rejects it with a stated reason, and the reasoning is written to the record either way.
The reviewer runs on a different model family from the agent it reviews. The argument for that separation is easy to state and harder to verify: a model auditing its own reasoning shares its own blind spots. It is an architectural bet, and we did not have evidence for it when we made it.
The finding: 65% disagreement, and it is not noise
The reviewer rejects 65% of the proposals it sees — 91 rejections across 139 verdicts, measured on 31 August 2026.
Those two numbers need their denominators stated, because a rejected proposal is re-proposed once with the feedback attached, so a single finding can produce more than one verdict. Against verdicts, the disagreement rate is 65%. Against findings that reached a decision, just over half are ratified: 48 of 90. Both are true, and reporting only the friendlier one would be the easier thing to do.
The first reaction to that number is that something is broken. A 65% disagreement rate looks like a defect rate, and the instinct is to tune it down until it looks reasonable.
That instinct is wrong, and it is worth understanding why. A reviewer that ratifies everything is indistinguishable from having no reviewer at all. The disagreement rate is not a defect metric — it is a health metric, and the direction that should worry you is the one approaching zero. A review layer that never objects is a review layer you are paying for and not receiving.
The rejections are not random
The rate alone would not be interesting. What makes it actionable is that the objections cluster. We categorized every rejection reason expecting a spread across many failure modes. Instead:
Severity escalated beyond what the cited CVSS evidence supports — 55% of rejections.
Remediation text naming no specific version or patch — 24%.
Proposed deadline conflicting with the regulatory due date — 23%.
Prompt-injection text found in the scanner's own comment field — 15%.
A single rejection can cite more than one of these, so the shares deliberately total more than 100%.
More than half of all rejections say the same thing. Verbatim, from the record:
"The severity is escalated to critical without evidence supporting such a jump from the CVSS base of 7.8, and the remediation is vague."
"The severity is rated as critical despite the scanner's CVSS being 5.4 and the NVD description indicating a high (7.8) severity, creating a mismatch between evidence and proposal."
That fourth category is worth pausing on: in 15% of rejections the reviewer was not disagreeing about severity at all. It was flagging that the scanner's comment field contained text instructing whoever read it to treat the finding as a false positive and close it quietly. The review layer was doing double duty as an input-integrity check.
The first model has a consistent bias toward inflating severity past the evidence it cited itself. Not occasionally, and not randomly — systematically, in more than half of the objections raised against it.
This is the part that deserves care, because it is easy to overclaim. We cannot prove that a same-family reviewer would have missed this. That experiment is one we did not run, and anyone telling you otherwise about their own system probably did not run it either. What we can say is narrower and still worth something: the cross-family reviewer caught the bias, caught it systematically rather than occasionally, and wrote down its reasoning in a form that could be counted and audited afterward.
Nobody found this bias by reading outputs. The architecture found it and filed it.
Why this matters outside vulnerability management
The specific domain is incidental. The pattern is not.
Any AI system that produces a judgment which becomes a record — a risk rating, a claim decision, a candidate score, a compliance determination — including one produced by a vendor's model rather than your own — has the same structural problem. If the only check on that judgment is the model that produced it, or a sibling of that model, then a systematic bias in its reasoning is invisible by construction. It will not show up in spot checks, because spot checks sample outputs rather than compare reasoning — the same blind spot that lets traditional security assessments miss AI risks entirely. It will not show up in accuracy benchmarks, because the bias is in justification, not in the answer being obviously wrong.
It shows up when something structurally independent is positioned to object, and its objections are recorded in a form you can count.
Three things made that possible here, and none of them are exotic:
The reviewer is independent by architecture, not by instruction. It is a different model family, from a different vendor lineage, reading the same evidence. It is not the same system asked politely to be critical. Worth being precise about the limit: separation of model family is what our measurements support. Running each agent under its own hardened runtime identity is a further step, and one worth taking, but it is not what produced the finding above.
Disagreement is recorded, not resolved silently. Every rejection carries a stated reason, written to a durable record. That is what turned a vague sense that "the model over-rates things" into a countable 55%.
The disagreement rate is a headline metric, not a buried one. It appears on the operational dashboard beside the counts it qualifies. A rate that is buried is a rate nobody notices drifting toward zero.
Two related findings worth naming
Restraint is a result. The system runs unattended on a schedule. One morning it woke, evaluated every open finding, and did nothing — the remediation deadlines run from seven to thirty days and none had come due. The correct autonomous answer that day was "not yet," and a system that manufactures activity to look busy is worse than one that waits. If your AI system has no observable state in which it correctly does nothing, it is not exercising judgment.
Models are not deterministic, and that has consequences beyond quality. Running the same evaluation twice produced different decisions for the same input — one item ratified on the first pass and rejected on the second. The usual argument for guarding against a repeated run is avoiding duplicate work. The stronger argument only appeared under test: without a guard, a repeated run would not merely have duplicated effort. It would have silently overwritten a recorded decision that a person may already have acted on, replacing it with a contradictory one.
Any system where an AI judgment becomes a durable record needs to answer that question explicitly. What happens when the same input is evaluated twice?
What to take from this
If you operate an AI system whose outputs become records, three questions are worth asking this quarter, alongside the five every executive should be asking about AI security:
- What independently checks this model's reasoning, and is it independent by architecture or only by intention? A second call to the same model is not a review layer.
- When the check disagrees, is the disagreement recorded in a form you can count? If disagreements are resolved silently, you have a filter, not a reviewer.
- Do you know your disagreement rate, and would you notice it trending toward zero? A review layer that stops objecting has usually stopped working, and it fails quietly.
None of this requires a larger model or a bigger evaluation budget. It requires positioning something structurally capable of objecting, and then keeping a record of what it objects to.
The model is rarely the hard part. The architecture around it decides whether you can trust what it produces.
Ayliea helps organizations assess and govern the AI systems they depend on. If you are deploying AI into decisions that become records, request a scoping call or read more about our methodology.