Technology · Hardware

NVIDIA Blackwell GPUs

Consumer NVIDIA Blackwell GPUs carried the Inference Fabric, the capabilities model and early ActiveKV work, first on one card and then on two mismatched cards draining one broker queue.

Helix, FAFO Core, Inference Fabric, ActiveKV, capabilities modelCompute capability 12.0 · CUDA 13.1A 32 GB and a 16 GB card
What the broker sees of each card
  • 96.9%streaming-multiprocessor utilisation on one card
  • ~65,000tokens a second held for ten hours on one card
  • 4.95Mtasks in that ten-hour run, with zero retries
  • 3–4 hto build the shared TensorRT-LLM image from source

I have run consumer Blackwell GPUs hard in five roles: bulk generation, a co-resident custom decoder, real-time speech, provisioned compute and early cache-offload work. In each I measured what the card does and built to that; cache windows, memory left beside a resident model and utilisation all come from the card.

Bulk generation on one card

Inference Fabric asked how much of a consumer Blackwell GPU's real throughput a disciplined serving design could reach on bulk generation. On one 32 GB card the 7-billion-parameter dense model ran in FP8.

The card
32 GB card
generation engine tiers and the analytics kernels share it
How the analytics engine reaches it
Generation runs on TensorRT-LLM engine tiers, one per card, each in its own image.

Two mismatched cards, one queue

Then a second, smaller 16 GB Blackwell GPU joined it, both draining one backlog.

Mismatched cards are where a plain queue breaks down.

The big card wastes capacity on small batches, the small card gets work it cannot fit, and batches that mix prompt shapes make the engine's cache thrash. I wanted to solve that once without teaching the scheduler about either card. Each GPU consumer declares four integers that describe its limits and its model, and the broker packs batches to those numbers knowing nothing else about the card. The smaller card runs its own engine tier built for its memory, in its own per-GPU container image.

Sequences per step 32 GB card · 512 to 4,096
16 GB card · 128 to 1,536
Tokens per step 32 GB card · 32,000 to 524,000
16 GB card · 16,000 to 98,000
The same 7B dense model with FP8 weights and FP8 key-value cache on both; only the engine changes.

I added the second card only after the single-card path was saturated, so its gain could be measured on its own, and single-card and two-card results are reported separately, each with its model, precision and window. The broker grouped work by token shape.

Five-minute mean · tokens a second
All tokens163,000Generated~69,000
0.42% padding waste99.9% key-value cache hits0 shape switches

See A message broker that understands tokens and From 44K to 163K tokens per second.

Built for compute capability 12.0

Consumer Blackwell is compute capability 12.0, and when I started on it the software had not caught up. I built NVIDIA TensorRT-LLM from source for the architecture, as one image both cards share, and compiled every analytics kernel with CUDA 13.1 as a native SM 120 binary with no PTX fallback. A runtime check refuses any GPU that is not compute capability 12.0 on a CUDA 13 driver, so a kernel never runs on hardware it was not built for. The kernels are sized for the card's shared-memory budget.

Lattice tile height
64 rowschosen128 rows
Threads in a fused search block
64 rows keeps register and shared-memory pressure steady within 8 to 24 KB of shared memory per block. Scalar FP32 throughout, leaving the tensor cores to the inference engine, because every analytics result must match a CPU reference.

Everything above the kernels is architecture-neutral, so datacentre Blackwell is a rebuild of binaries, the hardware check and an engine tier; the broker, the consumer contract and the provisioning stay as they are.

A router beside a resident model

Capabilities model had to live on a Blackwell GPU whose memory belonged to a resident model of about 30 billion parameters. I sized everything from measurements of the card itself. Full fine-tunes of a 0.6-billion-parameter open-weight base model finish in minutes on it, and the custom CUDA decoder was tuned to the cache the device actually provides.

The card's memory
resident model, about 30Brouter engine · 2,740 MiB
A burst of simultaneous clients
Served2,048Rejected0Corrupted0
Proportions illustrative. Work is admitted against measured memory, and a GPU that looks idle is checked against its compute processes first.

See A router beside a production model and Trying the industrial path first.

The same function on either card

FAFO Core and Function Forge has to run the same function on whatever GPU it lands on. I made the engine's token budget a required part of each node's description and had the batching layer read it from the node that runs the work. Each card is one more provisioned node with its own runtime image, and the same function packs batches correctly for either with no code change.

Node description, with its token budgetBatching layer reads itBatches sized for that card
Description with a budget
provisionedDescription without one
refused
One captured window, both cards: 876,000 probes submitted, 131 million tokens queued, 156 probes a second, zero retries.

See The workload that shaped the platform.

Speech, and the first cache offload

Helix needed speech fast enough for conversation on hardware I own. A Blackwell GPU hosts its the speech recognition model speech recognition and GPU media work.

One card over time, illustrative
Voice
Render jobs
Render jobs are admitted against voice, so a long render cannot take the card from a conversation in progress. Speech synthesis moved to a dedicated host after replies arrived late on a shared GPU.

See A self-hosted colleague in a Teams call. ActiveKV began on a Blackwell GPU too: a precursor patch to vLLM offloaded a hybrid model's attention cache to host memory on this card, and later analysis on the card produced stock reference runs for comparison. The hardware, drivers and TensorRT-LLM are NVIDIA's; the engines, kernels, per-card configuration, schedulers and admission rules that ran on them are mine.