Engineering story · Inference Fabric

Throughput that finished nothing

The runtime reported more tokens per second than it ever had while finished sequences fell below one per second; the fix gated new work on how many sequences were already alive

Rust and C++ runtimeInference FabricDense 7B engine
Full load, no admission limit 160,000+ tokens a second, the highest rate the dashboard had shown
The fix · a latch on active slots new prefills wait while the sequences already alive finish
<1finished sequence per second
  • 4,096sequences per step on the largest engine tier
  • 524,288tokens per step on the same tier
  • ~4,100sequences active when nothing was finishing

The first time the runtime ran the largest engine tier I built for the dense 7B model under full load, the token rate and the completion counter disagreed. The GPU was generating at full speed and finishing nothing.

What was happening

Continuous batching lets the engine admit a new sequence into any step that has room. With no limit on admissions, the runtime filled every slot, and each step then advanced four thousand sequences by one token each. Every sequence made progress, so the token rate was real, and every sequence made progress so slowly that none of them reached its end. The workload needed finished sequences, with their outputs written back, and a steady state of four thousand half-finished ones was worth nothing to it.

Every slot admitted

Each step moves every sequence one token. The token rate is real, and no sequence reaches its end.

New prefills gated

The same work goes to fewer sequences, which finish, write back and free their slots for the next ones.

Illustrative: progress of ten sequences in flight. Full bars are finished sequences.

The latch

The fix is a high-water and low-water latch on active slots.

Active slots over time
High-water mark: at or above it, new prefills are gated and only decodes run
Gated: the sequences already alive finish and free their slots; at or below the low-water mark, prefills resume
Prefills admitted

The two thresholds come from the provisioner with the engine, both or neither, so an engine cannot be provisioned with half a policy. A second defect was fixed in the same pass: finished slots are now reclaimed before every allocation, which ended a starvation in which allocation failures spiked while one to three slots stayed free.

Who enforces what

The batching contract is split between the two halves of the runtime. The token budget rule has a regression test, because an earlier version let a budget squeeze out sequences that were already mid-generation.

One step
Continuing decodes: always included, outside the budget
New prefills: the only part the token budget limits
The Rust side enforces the minimum batch, priorities, fairness and the slot budget; the C++ side enforces the engine's maximum and does the final packing.

Behind the admission rule is a ladder of backpressure in the decode loop.

Retry buffer, served firstBroker pulls, bounded by free slotsBuilder stops at a pending ceilingFeedRelease tokens
With no new work, a progress-only step still advances active sequences; an oversized item fails back to the broker with empty output rather than stalling the consumer.

What this demonstrates

Choosing the right metric.

Tokens per second is what an inference runtime advertises, and under a batch-oriented workload it can be maximised while delivering nothing.

The latch trades a little peak rate for sequences that finish, and the runtime's two-card figures were measured with it in place.

The latch and the split of batching duties were my design; AI coding agents, a mix of frontier and open-source models, implemented them under my governance system.