Corey JamesEvery performance programme I have run has produced at least one result that was really a property of the measurement. A profiler that serialised the copy engine it was observing. A test suite that passed a thousand trials against the wrong code. A benchmark whose baseline had a stall bug. A headline that beat a weak comparison. I treat the instrument as part of the experiment, and I do not act on a number until I know what the instrument did to it.
The shape
Every instrumented run has an uninstrumented control. Every evaluation set is frozen before the model sees it and carries known-answer controls that can catch the team as readily as the model. Every "switched" claim about a code path comes with proof of binding order on the real startup path. Every baseline is the strongest one I can build, and the one that produced a flattering number is never used.
Where it appears
In ActiveKV, a confound experiment showed that timing-enabled events were lowering into timestamp commands that forced the copy engine to wait, so the serialisation the profiler reported was the profiler's. I retracted the verdict in the plan: The observer that changed the answer. In the same programme, over a thousand passing trials were voided when the native engine turned out to be bound after the selector that checked for it: Proving a rewrite is the same system.
In the Capabilities model, the strongest classical retriever I could build tied the model: The baseline that tied us. The decoder's determinism is checked by gates across batch compositions and by a cross-check against an independent implementation: Bit-identical across batch compositions.
In the Inference Fabric, the thirty-minute run with a stall bug is excluded as a baseline, and a diagnosis of a write path was done from the database's own tuple statistics because the queue's counters measure intake: 196 deadlocks to zero.
In AgentOS, the principle is institutional: reviewers never match hashes, because hashes let them stop reading; they reason about content.
What it buys and what it costs
It buys results that survive. It buys faster programmes in the long run, because a wrong verdict acted on costs more than a control run.
It costs time at every step, and it costs the occasional public retraction. I count both as cheap.
The judgment in it
The instinct to protect is the instinct to believe a number that agrees with you.
The habit I have built is to ask, before acting, what else could have produced this reading, and to run the variant that would tell me.