Corey James- 44Ktokens per second from the clean engine sweep, the baseline
- 4.95 Mtasks in a ten-hour run on one GPU, zero retries
- 96.9%streaming-multiprocessor use of the larger card in the two-card window
- 3plausible ideas measured and dropped
Stock TensorRT-LLM, set up the obvious way, left most of a consumer GPU idle on bulk generation. I was building a pipeline that would generate and mine tens of millions of items, so idle capacity was the whole cost. I took one model, a 7-billion-parameter dense model in FP8, from about 44,000 tokens per second on a single NVIDIA Blackwell GPU to a five-minute mean of 163,000 across two NVIDIA Blackwell GPUs, with no custom model kernels. Each step was a change to one layer of the serving stack, measured before and after, and the changes that made things slower are on this page next to the ones that worked.
The challenge
The ceiling was never the engine. It was everything around the engine: how work was fed to it, how batches were shaped, how producers and GPUs were coupled, and how many cards could share one queue.
Constraints
The same model throughout, so the comparison means something. Consumer hardware, a Blackwell GPU and later a second, smaller Blackwell GPU, with no datacentre interconnect. No custom model kernels. And a dated capture behind every number.
A throughput figure without its window and hardware is a rumour.
My approach
Change one layer at a time and measure it. I worked through four layers in order, with a dashboard that made each layer's effect visible, and I kept a table of everything that did not help.
What I built
An engine tier sweep that found the largest single engine-side lever, the token budget the engine is allowed per step.
Producer decoupling, so the processes generating work are separated from the engine's consumption and a slow producer can never starve the GPU.
A shape-aware broker that groups requests by token geometry so batches waste almost no padding and the key-value cache is reused instead of rebuilt. The broker is its own story, A message broker that understands tokens.
Cache-coherent dispatch, keeping related work on the same engine so that shared prefixes stay hot.
A second GPU on the same queue. Once the single-card path was saturated, a smaller Blackwell GPU joined the first as a second consumer, with the broker balancing between them.
Dashboards that made every step visible and gave me the captures this page is built from.
Difficult decisions
Rejecting three plausible ideas on evidence. Each was tried, measured, written down and dropped.
The centre line is FP8 on stock kernels. INT4 weights were also slower than FP8 on this hardware.
Choosing the baseline honestly. A thirty-minute run on an early version of my own runtime had a stall bug. I do not use it as the baseline, because a depressed starting point would inflate the improvement. The baseline is the clean engine sweep.
Results
The long-prompt run briefly touched readings above 250,000 across the two cards, and I report the thirty-minute mean. A separate ten-hour run on one Blackwell GPU held about 65,000 tokens per second.
Technologies used
TensorRT-LLM · FP8 quantised inference · CUDA profiling with Nsight Systems · Prometheus and Grafana · Rust and C++ services · Token-geometry-aware batching · NVIDIA Blackwell GPUs
What this demonstrates
A fixed model, one variable at a time, a capture for every number, and the failures published beside the wins. A single card's throughput doubled and a second card took the pair to 163,000, without touching a kernel. That is what disciplined systems work above the engine looks like.
TensorRT-LLM and the GPU drivers are NVIDIA's; the broker, runtime and dispatch design are mine.