Corey James- 7.5 GBengine footprint on disk, every tier
- 3.5 sengine load time
- +56%throughput at a 70 percent shared prefix on the larger card
- 6lines changed in the vendor model graph
A built inference engine is a commitment to one GPU, one precision and one set of limits. It will load on that card and refuse any other, it holds exactly the batch and token budget it was built for, and it carries the key-value cache layout it was given. The fleet that reached 163,000 tokens per second in From 44K to 163K tokens per second ran on engines I built this way, a dozen of them from one checkpoint, each sized to the card in front of me.
One checkpoint, many engines
The model is a 7-billion-parameter dense instruction model. I quantize it once: FP8 weights and an FP8 key-value cache, calibrated, with the output head left in high precision. Every engine is then built from that one checkpoint, so the engines differ only in their limits and cache layout. They share paged key-value memory and FP8 context attention. The base image they run in was built from source for consumer Blackwell only, and the model graph in it carries one small change of mine, described below.
Tier by token budget, then by batch
Sweeps established that tokens per step, the number of tokens the engine processes in one forward pass, is the dominant lever on these cards. So tiers double the token budget and scale the batch alongside until the next tier runs out of memory. On the larger card, with 32 GB of memory, the tiers climb to 4,096 sequences and 524,288 tokens per step. On the smaller card, with 16 GB, they stop at 1,536 sequences and 98,304 tokens, with a halved variant at 49,152 kept for headroom.
Most tiers cap each sequence at 1,024 input tokens and 128 output tokens, which keeps every sequence's cache small and the number of slots large. Separate long-sequence tiers accept 3,840 input tokens with larger cache blocks, for the prompts that need them.
The contract travels with the engine
The broker and the provisioner read it, and the runtime refuses to load an engine without one. A node is configured by the engine it loads, which is what lets One call to a GPU anywhere start a consumer anywhere with an engine path and nothing else.
Caching measured on the real prompt structure
My workload shares long prefixes across prompts, so I measured what block reuse was worth at the geometry I actually run: batch 768, a 98,304-token engine, 128 tokens in and 128 out.
Those numbers are why the broker groups batches by prefix.
Ideas tested and dropped
Several plausible ideas were benchmarked and rejected.
Also dropped: 4-bit weight-only for bulk work, slower than FP8; two executors per card, each loading its own 7.2 GB of weights. The hybrid recurrent-attention model could not be built into an engine and went to ActiveKV.
Six lines in the vendor graph
For hidden-state extraction I needed the representation after the final normalisation and before the output head. The only change I carry in the vendor tree marks that tensor as an extra engine output, so every decoder model built from the tree exposes it without altering the inference path, and the runtime validates its shape at load. From there the extraction engines strip the model down: 128-token inputs, one generated token, then no key-value cache and no generation at all, at batches up to 8,192 sequences and a million tokens per forward pass. The workload those engines serve is in The workload that shaped the platform.
What this demonstrates
Treating an inference engine as an engineered artifact with a target, a precision, a set of limits and a contract, choosing those limits by measurement, and publishing the ideas that lost alongside the ones that won.