Corey James- 62,400items dispatched and unresolved when the ring stalled
- 0pending tokens at that moment, with the GPU idle
- 27tests covering the event pool's checks
Batches reach the GPU through a hot queue in pinned host memory: mapped, write-combined, pre-faulted pages holding a ring of batch descriptors, with a doorbell the producer rings to publish and the GPU waits on. No copies, no allocation after warm-up, and one accounting error that stopped two GPUs.
The deadlock
When a batch does not fit before the end of the ring, the producer pads to the wrap point and writes the batch at the start. Every wrap left a few ghost bytes allocated and unreleasable, and over a long run they accumulated until the ring reported itself full with nothing in it, with a second consumer starving behind the first. The fix was a padding field on the descriptor that the acknowledgement adds back. One field, and the invariant that what the producer takes the consumer returns.
Each wrap strands its padding. The last row is a ring that reports itself full with nothing in it.
One worker, one session
A second accounting error lived in the broker connection. Each GPU worker registered twice, once for batches and once for completions, which the broker saw as two consumers. Heartbeats arrived on one session while work was tracked on the other, the session with the work timed out, and working GPUs were declared dead and had their work reclaimed. The fix collapsed the two into one registration on one socket, with the rule that both halves fail together, so the broker's view of a consumer is one session that is either alive or gone. The broker's side of the same relationship is in Recovery that trusts only what happened.
Green dots are heartbeats; light blue spans are completions. After the fix both halves fail together, so a working GPU is never declared dead.
Streams and events, by rule
The same runtime keeps its CUDA streams on written rules that the code enforces rather than documents: synchronising a compute stream is refused at run time, any capture state on a managed stream is rejected, exactly one named function may synchronise everything and only at teardown, and events come from a pool with generation counters, bounds checks and an owner-thread check, covered by their own tests. Completions survive a broker outage: the worker keeps its finished work durably while the broker is away and hands it over when the connection returns.
What this demonstrates
Zero-copy and lock-free paths fail by accounting rather than by crashing, and the failures show up as a system that is quiet when it should be busy.
Both fixes were small, both invariants are now written down, and both appear as metrics.
The ring design and the single-session rule were mine; AI coding agents, a mix of frontier and open-source models, implemented the fixes under my governance system.