Engineering story · Capabilities model

Prove the mechanism before the model

Three load-bearing ideas behind the custom decoder, a device-side graph loop, a persisting L2 window and a constant-memory trie mask, were each proved on the real GPU in a small lab before the model existed, and one of them contradicted the documentation

CUDA C++Capabilities modelOne NVIDIA Blackwell GPU
Three labs on the real GPU · before the model existed
Lab 1 · graph loop
the GPU writes its own predicate
Lab 2 · L2 window
the card disagreed with its docs
Lab 3 · trie mask
served from constant memory
3mechanisms proved first
  • 4.304 µssaved per token per kernel by looping on the device
  • 60 MiBof the 96 MiB L2, the set-aside ceiling measured on the card
  • 0.016 µsto consult the constant-memory mask at a 131,072-token vocabulary

The custom decoder for the tool-discovery model, told in Constraint made cheaper than no constraint, rests on three mechanisms that the whole design assumes will work: a decode loop that runs on the device under a graph whose continuation predicate the GPU itself writes, a window of L2 cache set aside so the model's weights stay resident between tokens, and a trie-shaped grammar mask served from constant memory. If any of the three had not worked on the generation of GPU I had, the decoder would have been built on sand. So before the model was trained, each was built and measured on its own, on the real card, with a toy body.

Each mechanism got a small kernel pair on the target GPU and one number before anything was built on it.

The device graph loop

The first lab kernel pair runs a decode step inside a graph with a conditional node: the step writes a predicate, and the graph loops while the predicate holds, with no host round-trip per token.

Host-driven loop

A round trip to the host between tokens.

Graph loop on the device

The step writes its own continuation predicate.

That saving is what justified the graph-captured step in the decoder, and it is why the decoder's step writes its own continuation predicate.

The L2 window

The second lab pits a kernel that reads the weights against an antagonist kernel that thrashes the cache, with a persisting-access window set aside in L2. The documentation suggests a ceiling for the set-aside, and the card disagreed; a request above the real ceiling is refused by the driver rather than clamped.

Share of 96 MiB of L2 that can persist
Documentation75%Measured62.5%
The decoder sizes its window to the measured ceiling and reports the refusal if it is exceeded.

The measured gain for batch-one decode was about one percent, and the decoder's notes keep it as a fact, withdrawn as a lever.

The constant-memory mask

The third lab serves the grammar's token mask from constant memory and measures the cost of consulting it during sampling. That cost is the number behind the decoder's claim that the grammar constraint costs nothing measurable, and it was in hand before the first constrained token was ever sampled.

One decode step · tokens the grammar allows
constant memorythe trie node for this step, read once per sampled token
Illustrative slice of the vocabulary.

What this demonstrates

Separating the risky mechanisms from the system that depends on them, proving each on the target hardware with a measurement, and noting where the hardware disagreed with its own documentation.

The decoder was then built on three numbers rather than three hopes.

The choice of mechanisms and the decision to prove them first were mine; AI coding agents, a mix of frontier and open-source models, built and ran the lab under my governance system.