Corey James- 15hand-written CUDA kernel families, native Blackwell binaries, no JIT
- 0allocations after the first second of a profile, down from 4,236
- 4.95 Mtasks on one GPU over ten hours, zero retries
- 126Klines of first-party Rust, C++ and CUDA across six trees
Large offline model workloads are throughput problems. Synthetic data, probe generation, hidden-state extraction and bulk embedding analytics do not care about response time; they care whether the GPU is busy. A request-and-response serving stack wastes GPU time on padding and idle gaps and ties producers to GPUs for hours. I built the Inference Fabric for throughput. Producers drop millions of tokenised tasks and walk away. A broker that understands tokens organises the work so GPUs barely pad. Pull-based GPU workers stay saturated on mixed hardware. Results land in the database when they finish. Vector outputs flow into a GPU-native store that searches and scores cluster quality. On two NVIDIA Blackwell GPUs it ran a 7-billion-parameter dense model in FP8 at 163,000 tokens per second.
The idea
The token is the unit of scheduling. Work is grouped by shape so that each batch is shape-pure, packed to each GPU's declared limits, and dispatched so that shared prefixes stay hot in the engine's cache. A consumer describes itself with four numbers, and the broker packs for any card from those alone. See A message broker that understands tokens.
Nothing waits on anything else. Each service restarts independently, and work retires only when its completion is acknowledged, so a full channel delays an item instead of losing it and a restart never re-runs work the GPUs already hold. See Millions of queued items that were not there.
What I built
The buffer service, a durable broker on memory-mapped append-only logs with an NVMe cold tier, token-shape-aware grouping and per-consumer packing.
The saturated inference runtime, a pull-based GPU consumer around NVIDIA TensorRT-LLM for generation and raw TensorRT for single-pass hidden-state extraction, with no custom model kernels. Every gain in From 44K to 163K tokens per second came from the layers around the engine.
The completion service, which closes the loop for submit-and-exit producers and dispatches results per function into the governed write-behind data plane.
The vector shard manager: vector storage leased to the GPU that stays verifiably tied to its relational source, with index-quality reporting so retrieval health is visible, and only results returning to the database.
A CUDA analytics engine written from Rust against the driver API: 38 per-cluster health metrics and exact, inverted-file and lattice vector search, each kernel with a CPU reference and equivalence tests.
Federated GPU provisioning: one call stands up a consumer node locally, on a LAN host over mutual TLS, or on rented cloud spot capacity, behind a fail-closed daily dollar budget.
About 126,000 lines of first-party Rust, C++ and CUDA across six trees. TensorRT, the CUDA driver, PostgreSQL, Redis, Docker and the monitoring stack are third-party; the orchestration, protocols, storage formats, scheduling, analytics kernels and operating discipline are mine.
Same kernels, different supply chain
The runtime is a Rust and C++ saturation harness around stock TensorRT-LLM kernels, FP8 weights, cache and attention, and the gains come from workload supply: token-budget feeding, a ladder of backpressure in the decode loop, a zero-allocation hot path with pre-faulted pinned pools measured under Nsight (4,236 allocations in the first second of a sixty-second profile and none after), a sync discipline in which compute-stream synchronisation is physically rejected by the runtime, and a fail-closed engine loader that requires nine engine fields with no defaults. A failed worker exits fast and the orchestrator replaces it; one GPU runs one engine. The CUDA side is fifteen hand-written kernel families compiled to native Blackwell binaries with no PTX and no JIT, batched by compressed sparse rows, with a fused search-plus-health kernel. The codebase targets CUDA 13 and Blackwell SM 120, and its build also targets Ada and Hopper; the substrate it provisions reaches from Blackwell workstations to Hopper and datacentre Blackwell parts.
I did not replace NVIDIA's inference; I fed it.
Results
A 7-billion-parameter dense model in FP8 on NVIDIA Blackwell GPUs.
The two-GPU run peaked at 177,000; about 69,000 of its tokens a second were generated.
Zero shape switches, and a difference of zero.
The judgment in it
I measured three plausible ideas and rejected them, and they stay on the page next to the wins.
I took input-length filtering out of the broker because it stranded items and forced linear scans; the runtime fails oversized items back instead.
I kept the broker's hot path synchronous, one owner and a few blocking threads, because it was faster and easier to prove correct than an async design.
When six and a half million queued items turned out not to exist, I fixed the definition of "done" rather than the place the loss showed up.
What it demonstrates
Performance engineering with a fixed model and one variable at a time; broker and protocol design around the economics of the hardware; distributed-systems accounting that balances to zero; and hand-written CUDA where it earns its place.
Stories
From 44K to 163K tokens per second · Engines for the card you have · Every metric gets a CPU twin · Removing the sampling caps · Where a cluster sits · Diversity in three kernels · Physics as a lens on embeddings · Two-means as a measuring instrument · From health scores to decisions · Shapes have a lifecycle · The 128-lane wall · Done means delivered · Recovery that trusts only what happened · Starting full and ending clean · Throughput that finished nothing · A second engine behind the same door · Asking the engine what it meant · The thirteen percent that was never missing · Tests that passed without a GPU · Machinery I built and then did not need · Ghost bytes in a zero-copy ring · Twenty thousand rows a second · Pricing writes by what they cost the database · An index you can rebuild from its receipt · The database decides when an index goes stale · Rebuild only what moved · Three shards that move together · Storage that refuses to become a coordinator · The queue that should not exist · A doorbell the GPU waits on · A message broker that understands tokens · Millions of queued items that were not there · The workload that shaped the platform · Spend structure before spending waste · Thirty-eight metrics in a dozen passes · A detector for a silent corruption · Built for Blackwell
Technologies
Rust · C++ · CUDA C++ · CUDA driver API · Cluster-health metrics on the GPU · Streaming statistics in CUDA · Tiled pairwise kernels · CPU reference implementations and GPU equivalence testing · Gaussian-kernel cluster analysis · TensorRT-LLM and TensorRT · FP8 inference · Memory-mapped append-only storage on NVMe · Custom TCP wire protocols · Approximate nearest-neighbour search · Prometheus, Grafana and DCGM · NVIDIA Blackwell GPUs · Cloud GPU spot provisioning