Engineering story · Inference Fabric

Starting full and ending clean

A pressure gate holds dispatch until the first batches can be good, every batch closes on its token budget, and a cleanup mode drains the tail when the GPUs go quiet

Rust brokerInference FabricTwo-GPU run
One workload, start to finish
gate holds · tokens buildbatches fill the budgetcleanup drains
0items left in the tail
  • 523 µsaverage time to assemble a batch
  • ~100%batch fill at the median, the ninetieth and the ninety-ninth percentile
  • 4integers each GPU registers: batch size, tokens, input length, model

A batching broker is judged on its steady state, and its steady state is the easy part. The beginning and the end of a workload are where batches get small, caches get rebuilt and GPUs sit idle, and both needed their own mechanism.

The release gate

At the start of a run, items trickle in. A broker that dispatches the first trickle produces tiny batches and poor graph reuse, and the engine spends its first minutes doing badly what it could do well a moment later. So nothing is dispatched until staged and lane tokens together reach a release threshold.

Memory mode · the gate holds, then opens

Grey: staged and lane tokens building, nothing dispatched. Orange: the release threshold reached. Blue: dispatch from then on.

Disk mode · the gate is always open

The backlog is already on disk, so the first batches can be full from the first tick.

Once open, the gate stays open for the life of the process; it is a start condition rather than a throttle.

Batches that fill to the budget

Packing follows the contract each GPU registered with. In memory mode the padded length of a batch is the longest item rounded up to eight tokens and the budget is counted in padded tokens; in disk mode it is counted in actual tokens. Input-length filtering was removed from the broker after it stranded items, and the runtime applies that limit itself. The ready-batch search scans lanes round-robin from the last lane served for an active lane holding at least the first consumer's batch size.

What closed the batch
Token limitallItem limitnone

Each batch filled to its token budget before it reached its item count.

Assembly order
Reclaimed workDisk tierLanes

Work reclaimed from a dead consumer goes first, so recovered items are never starved behind new ones.

Dashboard figures from the two-GPU run.

On a single host, no copies

For consumers on the same host, batches go through a shared-memory ring on two-megabyte huge pages rather than a socket: a lock-free allocator with cache-line alignment and a compare-and-exchange loop to claim space. Batches are offset-addressed with no pointers, written with alignment safe for ARM, and mapped read-only by the consumer.

The ring, laid flat
Batches waiting: from the read pointer to the end, then from the start to the write pointer
Free space the writer may claim
Guard before the read pointer: the write pointer can never meet it
Tail gap: abandoned when the writer wraps to the start

Cleanup mode

At the end of a workload, staged shapes that never reached their activation threshold and lanes holding less than a full batch would otherwise sit forever.

The end of a workload, illustrative
normal batchescleanup: threshold 0, batch of 1empty
It begins when every dispatch channel is empty, work remains in staging or lanes and no normal batch is ready; it ends when staging and lanes are empty, or at once when new work arrives.

Entries, duration, and batches and items dispatched in cleanup are metrics, so a run that spends too long in its tail is visible.

What this demonstrates

Engineering the edges of a workload as deliberately as its middle: a start condition that guarantees good first batches, packing measured against its own budget rather than its item count, and an end state that drains to zero rather than leaving a tail behind.

The gate, the packing rules and cleanup mode were my design; AI coding agents, a mix of frontier and open-source models, implemented them under my governance system.