Engineering story · Inference Fabric

Shapes have a lifecycle

Every switch of prompt shape costs a GPU its prefix cache, so the broker stages a shape until it is worth running, warms the next one before the current one drains, and alarms when a shape ran too briefly

Rust brokerInference FabricTwo NVIDIA Blackwell GPUs
Active shape · lanes draining Next shape · warming before the switch
Graphs built ahead of time the GPU moves to the next shape with its cache ready
0shape switches, five-minute two-GPU window
Staged · active · draining · finished · warming
  • 5states every prompt shape moves through
  • 99.9%key-value hit rate in a five-minute two-GPU window
  • 100batches: a shape that finishes sooner is flagged as thrashing

A shape, in my broker, is a set of prompts that share the same ordered prefixes, which is what makes them cacheable together on the GPU. Switching from one shape to another throws away the engine's prefix cache and may force a rebuild of its CUDA graphs. A broker that interleaves shapes freely therefore pays that price constantly, and the engine spends its time rebuilding rather than generating. So shapes in my broker have a lifecycle, and the lifecycle is a state machine with thresholds, not a heuristic.

Five states

A new shape is staged. Its items accumulate per signature, and nothing is dispatched, because opening lanes for a handful of items wastes structure the engine could use. Items that arrive for a shape that is already active are merged into its existing lanes one by one; an early version created a fresh shape object instead and orphaned the old lanes with their data.

Staged
Staged tokens reach a multiple of the batch token budget
Active: new items merge into existing lanes
Pending tokens fall below the drain threshold
Draining: closed to new work
Finished: every lane returns its slot
While active, an item is assigned first and written second; late arrivals at a draining shape go back to staging.

Draining shapes never take new items, which is the invariant the whole design rests on.

Warming the next shape

The expensive moment is the switch, so the broker prepares for it. When an active shape's pending tokens drop below a pre-warm threshold, the broker picks the staged shape with the most tokens above the activation threshold and marks it warming. Only one shape warms at a time. Consumers receive a pre-warm signal naming the shape and lane, and can build the CUDA graphs for it ahead of time in a cache keyed by lane and shape epoch. Every packed batch carries its shape epoch and a homogeneous output-token limit, so the engine can size its key-value allocation for the batch before it arrives.

One switch
Current shape
Next shape
Active: the GPU is generating this shape
Draining (top) and warming (bottom) overlap: graphs are built before they are needed
When the old shape finishes, the warming shape activates immediately, so the GPU moves between cacheable workloads with nothing to rebuild.

The thrashing alarm

A shape that finishes after fewer than a hundred batches did not run long enough to pay for its own switch. The broker logs that as a critical event and feeds a batches-per-shape histogram, so a workload whose shapes are too small to amortise shows up on the dashboard as a pattern rather than as mysteriously low throughput.

Batches each shape ran before it finished
Shape 1
Shape 2
Shape 3
Shape 4
Shape 5
thrashing line
Illustrative run. Orange shapes are the ones logged as critical and counted in the histogram.

What the lifecycle replaced

The first design cut static lane boundaries from a per-shape token histogram: 512 piecewise bins, percentile splits with interpolation, gap detection over runs of empty bins. It worked and it was rigid; rolling lanes replaced the static boundaries, and the histogram survives as per-shape statistics copied onto the shape at activation. Lanes themselves are recycled when they stop earning their slot.

Items in the laneunder four
Seconds without an assignmentpast five
Seconds a lane has waitedpast thirty: starving
Bar lengths illustrative; the line is the threshold. A lane under the first two thresholds while another lane is active is force-drained and its slot goes back. Starvation is a metric, so it shows on the dashboard instead of as slow batches.

What this demonstrates

Treating the GPU's cache behaviour as a first-class scheduling constraint: the broker knows what a switch costs, stages work until the switch is worth it, prepares the engine before the switch happens, and reports when a workload defeats the design. The two-GPU result is told in From 44K to 163K tokens per second.

The lifecycle and its thresholds were my design; AI coding agents, a mix of frontier and open-source models, implemented them under my governance system.