Corey James- −6.8%host key-value offload, set aside as a rejected idea
- −7.5%class-aware batching on average, also rejected
- −36%class-aware batching at peak
CUDA graphs capture a sequence of kernel launches once and replay it without launch overhead, and a decode loop that runs the same step hundreds of thousands of times looks like the ideal candidate. Capture has a condition, though: the shapes must be stable. So I built the machinery that would make the runtime's decode steps capturable.
What was built
An epoch is a span of steps with a fixed number of active slots. Slots are dense, so a batch of a given size always has the same layout; the map from sequence to slot is a bit vector with constant-time lookup; and a change in key-value-cache bucket falls back to eager execution for that step without ending the epoch. The design was correct, tested, and ready for graphs to be captured per epoch and replayed.
Occupied slots packed together, no gaps.
One bit per slot: which ones hold a sequence.
What the profile said
Before wiring capture in, I profiled graph execution against eager execution on the same engine.
The vendor's executor already captures and replays graphs internally, and the launch overhead I was going to remove had been removed before I started. The same profile, in The thirteen percent that was never missing, is the one that corrected a misread utilisation figure.
What stayed
The epoch and slot machinery stayed, because it had a second job: slot integrity. A fixed slot count per epoch, dense slots and a constant-time map are what make the slot-gating latch in Throughput that finished nothing cheap and exact. Graph capture was dropped as a lever, and the runtime's notes set the result beside the other rejected ideas.
The first two rows stayed because the slot-gating latch needs them; the third was never wired in. Host key-value offload, two executors per card and class-aware batching sit beside it in the runtime's notes.
The rule
Read the library before reimplementing it, and measure before building the thing the measurement would justify.
The epoch work cost days and bought slot integrity; the graph work it was built for would have cost more and bought a tenth of a percent. The notes are kept so the next person who looks at that decode loop and thinks of CUDA graphs can read why not.
What this demonstrates
Engineering judgment applied to my own work: a sound design, a measurement that removed its reason, a decision to keep the part that still earned its place, and a written account of what was tried and why it stopped.
The decision to measure first and the disposition of the machinery were mine; AI coding agents, a mix of frontier and open-source models, built and profiled it under my governance system.