Corey James- 27gates audited in the eighth phase of the swarm build
- 2failed gates kept as failures, not narrowed away
- 1unrelated gate dropped by explicit decision
AI reviewers have two failure modes. Left alone they re-litigate forever. Drawn from the same model as the builder they share its blind spots and pass its mistakes. I applied the principle regulated industries use: the people who build do not judge, the judges are independent, and a failure stays a failure until it is fixed. In my system the builders, the quality reviewers, the gatekeeper and the adversarial reviewer are separate agents, the reviewers run on a different vendor's models from the builders, and a phase with two failed gates is held at a conditional pass with two failures.
The challenge
Agent teams report success. They are built to complete tasks and they complete the task of saying so. For the output to be trustworthy, something independent has to check it, and that something has to finish: an AI reviewer with no bounds finds a new concern on every pass. The human, me, had to remain the only override, and the override had to be rare enough to mean something.
Constraints
Review must be genuinely separate from building and must converge. The human can narrow or drop a criterion; nobody else can. Criteria that are wrong get dropped explicitly, by decision.
My approach
Separate seats with separate jobs. Builders deliver. Quality reviewers check against the acceptance criteria. A gatekeeper accepts only delivered facts, never promises. An adversarial reviewer on a different vendor's model looks for what the others missed. A final closer confirms nothing in the close evidence is missing. Follow-up review is scoped to the findings it was opened for so it converges, and a FAIL remains a FAIL until the fix is delivered and re-reviewed.
What I built
Governed follow-up review, scoped to the findings it was opened for.
Pressure checks and envelope audits that confirm a delivery stayed inside what was authorised.
A written review doctrine that the reviewer seats operate under.
The origin practice: gatekeeper reports written by agents who had never implemented what they judged, and closure packets that carried every gate's verdict.
Difficult decisions
No new role for testing. When the team proposed a dedicated testing role, I ruled that testing is a developer activity that produces evidence.
Adding a role would have let builders stop testing.
Scrubbing checksums from reviewer packets. Reviewers were matching hashes instead of reading content. I removed the hashes so they had to reason about what was delivered.
Holding a phase at conditional pass. The eighth phase of the swarm build audited 27 gates. Two failed: workspaces were not archived, and a working document contained credential-shaped values. The tempting fix was to narrow the criteria. I set the conditional pass with both failures on it, dropped one unrelated gate by explicit decision, and had the archival tooling built afterwards.
Results
Envelope review was delivered end to end through managed reviewer seats, with reviewers on different vendors' models from the builders.
Technologies used
PostgreSQL · JSON Schema · Frontier coding models from more than one vendor · Model Context Protocol · Review doctrine as a written artefact
What this demonstrates
Quality-engineering separation of duties applied to AI-produced work, and integrity in acceptance: the failures stay because they happened.