Corey James- 0.421%padding waste across one shared backlog
- 0shape switches mid-stream
- 24,000lines of Rust in four crates
A general-purpose queue moves bytes and knows nothing about what they cost to run. GPU inference cost depends almost entirely on the shape of the work: how long the prompts are, how much each prompt shares with its neighbours, how much padding a batch carries. I built a broker whose unit of scheduling is the token. It organises incoming work so that each batch is shape-pure, packs batches to each GPU's declared limits, and delivers them so the engine's cache keeps hitting.
The challenge
I had a backlog of billions of tokens to generate and two GPUs of different sizes to do it with. A conventional queue would hand each card whatever came next. The large card would waste capacity on small batches; the small card would be handed work it could not fit; batches would mix prompt shapes and the engine's cache would thrash. The ceiling was in what the engine was being fed.
Constraints
Nobody waits: producers never block on the GPUs and the GPUs never block on producers, and a backlog of billions of tokens survives restarts on disk. The broker must not know the model; it schedules work it cannot interpret. Different engines run on each GPU. And the hot path is the product, because every microsecond in the dispatch loop is paid billions of times.
My approach
Make the token the broker's unit and reduce each GPU to a contract. A consumer declares four numbers: maximum batch size, maximum tokens per batch, maximum input length, and the model it serves. From those the broker can pack a batch that fits any card without knowing anything else about it. Incoming work is organised so that a batch drawn for a card is shape-pure: the same prefix, similar lengths, minimal padding, maximal cache reuse.
What I built
A four-crate Rust broker of about 24,000 lines with adaptive packing per consumer from the four-number contract, and a guard that refuses to mix shapes in a single batch.
A disk-backed, shape-pure backlog on memory-mapped segments, so that billions of queued tokens survive a restart and are read in the order the GPUs want them.
Per-consumer dispatch threads with a synchronous hot path: a single-owner actor and blocking threads, chosen over an async design because the loop is small and hot and easier to reason about without a scheduler in the middle.
Per-GPU engine tiers, so each card runs the engine configuration that suits its memory.
Accounting panels that show padding waste, cache hit rate, shape switches and throughput per consumer in real time.
Difficult decisions
Taking length filtering out of the broker. An early version filtered oversized items before dispatch. It stranded items that fit no consumer and forced linear scans of the backlog. I removed it: the runtime fails an oversized item back, and the broker never scans.
A math-only contract. It would have been easy to teach the broker about each card by name.
Four integers describe any consumer, including ones that do not exist yet.
Synchronous on purpose. The fashionable choice was an async runtime. One owner, one loop and a few blocking threads was measurably faster and much easier to prove correct.
Results
Serving a 7-billion-parameter dense model in FP8 on two NVIDIA Blackwell GPUs:
Technologies used
Rust · Custom TCP wire protocol · Memory-mapped segment storage on NVMe · Token-geometry-aware batch scheduling · TensorRT-LLM · Prometheus and Grafana · NVIDIA Blackwell GPUs
What this demonstrates
I design infrastructure around the economics of the hardware it feeds, and I abstract at exactly the level the problem needs: four integers and a shape-pure backlog, and two mismatched consumer cards behave like one well-fed machine.