Engineering story · Inference Fabric

The 128-lane wall

Padding waste ran under one percent until every shape used all of its 128 lanes, then climbed past 95 percent on a 22-million-item load test; the fix was structural

Rust brokerInference FabricTwo NVIDIA Blackwell GPUs
One shape · 128 lanes, one byte of id
The least-bad lane
the item, then padding to the lane's length
95%+padding waste at the wall
  • 10Btokens in the 22-million-item load test that found the wall
  • 0.421%padding waste in the two-GPU window after the fix
  • 8phases of changes to widen the lane id

My broker packs prompts of similar length into lanes so that a batch pads as little as possible. Each lane has a target length, items go to the lane that wastes the least, and a new lane opens when the waste would be too high. The first version gave each shape 128 lanes, because the lane id was a byte and 128 seemed like plenty. A load test showed exactly what plenty was worth.

The curve

As the run went on, every shape reached a hundred percent lane pressure, meaning every one of its 128 lanes was in use and no new lane could open. The broker was doing exactly what it was designed to do: with no lane free, it assigned each item to the least-bad existing lane, and the least-bad lane was eventually terrible. Throughput did not fall gracefully. It collapsed, because the GPU was spending nearly all of its work on padding.

Padding waste through the run · 22 million items
Early0.5–1%Middle13%Lanes full45%65%76%wall
Once every shape had all 128 lanes in use, each new item went to the least-bad full lane, and the least-bad lane kept getting worse.

Why it was structural

The assignment algorithm was choosing correctly among the lanes it had. The budget was the problem. A workload with ten billion tokens has far more than 128 distinct length clusters per shape, and once the lanes were exhausted the algorithm had no good move left.

One shape · late in the load test
Length clustersfar moreLanesall in useLanes freenone
Illustrative proportions.

Tuning the waste threshold could not help, because the threshold only decides when to open a lane, and no lane could be opened.

The fix

The lane id became sixteen bits and the budget grew with it. With the budget raised, the threshold logic could do its job. Lane targets only grow, so an item never lands in a lane that has shrunk beneath it.

Waste allowed in an existing lane
Lanes plentiful
Two-GPU window
Lanes running out

One percent plus nine percent scaled by lane pressure: strict while lanes are plentiful, relaxed as they run out. In the two-GPU window it sat around three percent.

A new lane opens when waste exceeds
The threshold and 16 tokensOr 128 tokens outright
Lanes per shape, by lane id width
8 bits12816 bits2,048

The result

The budget is large enough that pressure never approaches the wall, and the threshold stays near its strict end.

Average lane pressure
At the wall100%Two-GPU window22%
Lanes reused against lanes opened, per second
Reused17,800Opened0.5

About 2,280 rolling lanes active across the shapes.

Dashboard figures from the two-GPU window; per-second rates are approximate.

What this demonstrates

Reading a performance collapse correctly. The symptom was waste, the tempting fix was the waste threshold, and the cause was a capacity limit that the algorithm could not work around. Raising the budget by an order of magnitude and letting the pressure-scaled threshold operate turned a collapse into a sub-one-percent steady state, told alongside the broader spend analysis in Spend structure before spending waste.

The diagnosis and the decision to widen the budget were mine; AI coding agents, a mix of frontier and open-source models, implemented the programme under my governance system.