Engineering story · Inference Fabric

Tests that passed without a GPU

Stub builds returned fake tokens and every test went green; the build now refuses to produce a binary without the real engine library, and the generation contract was fixed at the engine boundary before a corpus was reused

Rust · C++TensorRT-LLM runtimeNVIDIA GPUs
Test suite · run against the stub
GPU · never touched
engine library absent, stub linked
1000 + idthe fake token the stub returned for every sequence
  • 65,462tokens in a rejected batch, under a 65,536-token limit
  • 148candidate sequences in that batch, nothing out of bounds
  • 159 Mtokens dispatched upstream against an expected eight million

The inference runtime links a C++ library against the vendor's engine, and for a while the build could substitute a stub when the library was absent. The stub returned a fake token for every sequence, and the test suite passed against it. Two further bugs in the same period made the same point under real load.

A green suite that has never touched an engine is worse than no suite, because it says something true about the harness and nothing about the runtime.

The build that refuses

The build script now refuses to produce a binary unless the real engine library is present, and stubs must be requested by name. The integration tests abort if they find themselves running against a stub. That turned a silent substitution into a loud one, and it is the reason every figure on the runtime's pages came from the hardware it names.

At build time
Engine library presentbinary builds
Engine library absentbuild stops
Stub requested by nameintegration tests abort

Rejected batches

Under real load the vendor's executor rejected every batch with an invalid-argument error. Two contract details were wrong. Both were fixed at the request-building boundary in the C++ half, where the vendor's contract is translated, so no caller has to know either rule.

What a caller writes
promptend idtemp 0
What the engine is sent
promptend idpad idgreedy

The dark line is the request-building boundary in the C++ half. A pad id must accompany every end id, and greedy decoding is a sampling mode of its own, not a temperature of zero.

Garbage after the JSON

The generated probes are a replay corpus: they are stored and run again, so what is in them matters more than how they were made. A sample showed text after each JSON object. Both causes were fixed before the corpus was reused. Had they not been, every replay would have carried the contamination forward.

One generated probe, as stored
Before
After
The JSON object the probe should be
A literal end-of-sequence marker
Fragments of the prompt
Two settings cut the tail: stop on the end-of-sequence token, and leave the input out of the output.

Over-dispatch upstream

One more defect sat above the runtime. The conductor that feeds the broker was over-dispatching, because of a budget function that never ran. The fix lives in the orchestration layer; the account lives beside the runtime, because the runtime is where the symptom appeared.

BudgetDispatched
The budget function was imported and never called.

What this demonstrates

The difference between a passing test and a proven system. The build, the request contract and the corpus each had a way to look correct without being correct, and each got a check that fails loudly at the boundary where the truth is known.

The fail-loud build rule and the corpus decision were mine; AI coding agents, a mix of frontier and open-source models, implemented the fixes under my governance system.