Corey James- 0.42%padding waste across two NVIDIA Blackwell GPUs
- 99.9%cache hit rate on the same pair
- 12.3 Btokens received, balanced against the same number routed
Queues move messages. Schedulers count jobs. Budgets count dollars or requests. None of those is what a GPU pays in. A GPU pays in tokens, and its cost for a batch depends on the shape of those tokens: how long each sequence is, how much of each prefix is shared with its neighbours, how much padding the batch carries, whether the cache already holds the work. Once I made the token the unit of scheduling, packing, budgeting and accounting, a whole class of inefficiency became visible and fixable.
The shape
Work is described to the scheduler by its token geometry. Batches are packed to a consumer's declared token limits, not to a count of items.
Every row is padded to the longest; the light grey is paid for and wasted.
Same prefix, similar lengths: padding near zero, and the shared prefix, dark, stays in the engine's cache.
Budgets for a function are stated in tokens per step. Accounting counts tokens received, routed and completed, so a leak shows up as a difference between two large numbers instead of as a vague slowdown.
Where it appears
In the Inference Fabric, the broker's unit of scheduling is the token, and it packs for any card from what the consumer declares: A message broker that understands tokens.
In FAFO Core and Function Forge, token-aware batching is one of the three batching modes any function can select by configuration, with memory-budget packing beside it. The probe-generation pipeline is where it was forced into the substrate, in The workload that shaped the platform.
In ActiveKV, the token is the unit of the economics: a compressed representation is judged on bytes saved per token against GPU time added, and instrumentation follows each token from scheduler to device to client.
In the Capabilities model, the whole project is a token budget.
What it buys and what it costs
It buys saturation on mixed hardware, because packing by tokens lets a 16 GB card and a 32 GB card drain the same backlog at full tilt. It buys accounting that can balance. It buys honest cost comparisons between designs, because bytes and tokens do not lie about where time goes.
It costs a scheduler that has to understand shape, which is more code than a queue that moves bytes, and it costs producers that must tokenise before they submit. For chat-shaped traffic where each request is its own world, the shape of a batch matters less and a simpler scheduler is fine.
The judgment in it
The trick is abstracting at exactly the right level.
The broker must not know the model; it must know only what the consumer needs to pack well, and four integers turn out to be enough.
Everything above that is the engine's business and everything below it is bytes.