Corey James- 0.42%padding waste across the two cards, with zero shape switches
- 99.9%engine cache hit rate, from shape-pure batches
- 4.95 Mtasks on one GPU over a ten-hour run, zero retries
Offline workloads, synthetic data, extraction, bulk embeddings, are throughput problems, and a serving stack built for chat treats them like chat: padding, idle gaps, producers tied to GPUs for hours, results persisted by whoever is still waiting.
I took one model from about 44,000 tokens per second on one NVIDIA Blackwell GPU to a five-minute mean of 163,000 on two NVIDIA Blackwell GPUs without touching a kernel, by building the layers around the engine. Start with From 44K to 163K tokens per second.
I did not replace NVIDIA's inference; I fed it.
The first layer is a broker whose unit of scheduling is the token, feeding pull-based workers that stay saturated on mixed hardware. See A message broker that understands tokens.
Four numbers and nothing else; heights illustrative. The broker never learns a card by name and never reads the model; it packs any card from the contract alone.
One checkpoint, an engine tier per card. A slow card slows only itself.
Results land when they finish, through a completion service that closes the loop for producers that submit and exit, and the accounting balances to zero. The pipeline populated tens of millions of work items on two consumer GPUs. See Millions of queued items that were not there and The workload that shaped the platform.
An item was consumed the moment it was read and dropped if the consumer's channel was full.
tokens; difference zero
A full channel now delays an item instead of losing it, and a restart never re-runs work the GPUs already hold.