Corey James- 163Ktokens a second, five-minute mean across two NVIDIA Blackwell GPUs, from about 44K on one
- 3,840input tokens in the long-prompt tiers
- 1,000×the worst weight corruption the vendor conversion caused on affected tensors
Engineering everything around the engine
Inference Fabric started from stock TensorRT-LLM set up the obvious way, which left most of a consumer GPU idle on bulk generation. My view was that the ceiling sat around the engine, so I kept NVIDIA's kernels and engineered everything else. My C++ layer drives the engine through its Executor API from a Rust service and refuses any engine configuration that leans on defaults, so a mis-sized engine is refused up front.
I changed one layer at a time on a 7-billion-parameter dense model in FP8. An engine tier sweep found the per-step token budget to be the largest lever on the engine side, and it is the first lever I measure since. See From 44K to 163K tokens per second.
An engine tier for each card
Each card runs its own engine tier, all built from one calibrated FP8 checkpoint with FP8 weights, an FP8 key-value cache, paged context attention and FP8 context attention. On the 16 GB card a halved-budget variant adds headroom. Most tiers cap sequences at 1,024 input and 128 output tokens so per-sequence cache stays small and slots stay many, and separate long-prompt tiers take longer inputs with larger cache blocks.
Each engine comes with a contract carrying its limits, cache memory fraction and host cache size, so a node is configured by the engine it loads. I built the TensorRT-LLM base image from source for the cards' Blackwell architecture.
Grouping work by shared prefix
Before the broker existed I measured what shared prompts were worth, and that is why the broker groups work by prefix.
Admission control around the batcher
After I measured a case where the GPU stopped making progress with many requests in flight, I wrapped the engine's in-flight batcher in admission control that retries first and takes a progress-only step when the batcher stalls. Four other ideas were measured and dropped; a second executor on one card, for one, reloaded the full weights with nothing shared.
After a retry, the navy block is the progress-only step that gets the batcher moving.
- Host offload of the KV cachedropped
- INT4 weightsdropped
- Two executors on one carddropped
- Custom CUDA graphsdropped
Trying the vendor path first for the capabilities model
Capabilities model needed to serve about two thousand concurrent clients, and a custom engine is expensive to justify. I directed that TensorRT-LLM be tried first. Its guided-decoding extension point was traced in source and probed with the real catalog constraint, the model was converted with the vendor's tooling and compared with the source weights, and a throughput ladder ran in the vendor's container with and without the primary model resident.
I close a vendor path only on findings I can show, and three closed this one. See Trying the industrial path first.
An engine as an input to provisioning
FAFO Core and Function Forge treats TensorRT-LLM engines as inputs to provisioning. The batching layer reads the token budget of the engine on the node that runs the work, and provisioning refuses a node whose description leaves it out, so the same function runs correctly against either card's engine.
TensorRT-LLM and its kernels are NVIDIA's; my change to it is one marked output in the model graph that exposes the hidden state for extraction. The runtime, engine tiers, admission control and the vendor-path study are mine.
I get the most from NVIDIA's serving stack by engineering what surrounds it, and I judge it fairly when it does not fit.