Engineering story · Inference Fabric

A doorbell the GPU waits on

Batches cross from the CPU to the GPU through a pinned ring and a 64-byte doorbell the GPU's stream waits on directly, with no CPU synchronisation; and why that path is used for the extraction engines and left off for the vendor's executor

C++ · RustCUDA driver stream operationsNVIDIA GPUs
Pinned ring · written by the CPU filler
The doorbell
GPU stream
waits for sequence ≥ n, then runs
64 Bcache-line aligned
  • 1 : 1one producer, the filler thread; one consumer, the decode thread
  • 10retries of 1 ms on a full ring, then a block; a batch is never dropped
  • 0host threads between a publish and the GPU starting work

The batch filler writes a batch into memory the GPU can read directly, and the GPU's stream begins work the instant the batch is published, without a host thread noticing and launching anything. The CUDA driver has a primitive for exactly this, a stream operation that blocks until a 64-bit value in memory reaches a threshold, and the runtime's hot queue is built around it.

The fastest handoff from a CPU producer to a GPU consumer is one in which neither side waits on the other's clock.

The ring and the bell

Batches live in a ring of pinned host memory, mapped for direct GPU access and write-combined for fast CPU writes, with pages pre-faulted at warm-up so no first-touch fault lands on the hot path. The ring pads to the boundary so a batch is never split across the wrap, and its descriptors mirror the broker's packed-batch layout. Backpressure never drops a batch.

A batch never splits across the wrap
start of ringpadding to the boundary
Each descriptor · the broker's packed-batch layout
batch idlaneclassbucket maximumitem countreal and padded tokensoffsetsshape epochoutput limitpublish sequence
The shape epoch keys the graph cache; the publish sequence is what the bell carries. When the ring is full
bounded retriesthen block

The doorbell is a cache-line-aligned slot in mapped pinned memory holding a sequence number, the payload offset and length, and the batch id.

One handoff · two clocks
CPU filler
payloadfencebumpnext
GPU stream
wait: sequence ≥ nwork

Where it is used, and where it is not

The doorbell is the right design for the raw-TensorRT extraction backend in A second engine behind the same door, which reads its inputs on the device. The vendor's executor backend reads request data on the CPU, so for that backend the pinned path stays off. The accounting bug that once deadlocked the ring is in Ghost bytes in a zero-copy ring.

Extraction backend · pinned path on
GPU reads the ring directly
Executor backend · pinned path off
CPU reads request data a counter fires if the path is turned on here
Write-combined memory is fast for the CPU to write and slow for the CPU to read.

What this demonstrates

Using the driver's own synchronisation primitives to remove the CPU from a handoff, and matching the memory model to the consumer: a design that is right for one backend is stated as wrong for the other rather than switched on everywhere.

The handoff design and the backend rule were mine; AI coding agents, a mix of frontier and open-source models, implemented the ring and the doorbell under my governance system.