Corey James- 3.3%worst error of the bus latency model against measurement, 32K to 262K tokens
- 262Ktoken context on one card, at a 22 GiB device peak
- −6.8%throughput from KV offload on the NVIDIA serving path
Costing the bus before changing anything
ActiveKV: stock serving keeps a request's whole key-value cache on the GPU, so a 32 GB card caps how much context it can hold. I made pinned host memory the owner of the cache and gave the GPU a bounded working set. The exact path re-reads every byte of history for every generated token, so I wrote the cost down as a latency model before changing anything.
From then on, every design choice answered to that arithmetic. I treat the cache as a memory-hierarchy problem with a measurable cost on the bus.
I count bytes before I optimise.
Staging layers under a lease
Each staged layer holds a lease. History copies run on their own queue, attention waits on the copy's event on the device, and only the rows that changed go back to the host, on a second queue, after attention signals.
Each lease names the owner, the layer, the step, a stage generation that only increases and the host pages' generation. An unknown fence outcome quarantines the transport and keeps every buffer until a drain proves every copy has finished.
A new layer cannot start until the previous stage is sealed. A buffer is reused only after a finished wait on every copy and kernel that touched it, and whole-layer staging matches fully resident attention bit for bit. Exactness came first and speed was a separate programme.
Exact recent pages, compact older pages
For long histories, the planner cuts each request into tiles and into runs of two representations. Background threads convert a page only after it is fully written in every layer, compute a CRC and publish it. Until then the page is served exact, so decode never waits on conversion and a slow conversion only costs bytes.
Each tile's attention returns its normalised output and its log-sum-exp. An FP32 merge kernel combines them exactly: each partial is weighted by the exponential of its log-sum-exp minus the largest one, the weighted outputs are summed and divided by the summed weights, and the merged log-sum-exp is the maximum plus the log of that sum. That is softmax over the union of the keys; I never average separately normalised outputs. I patched a log-sum-exp output into the vendor's paged-decode kernel to make it possible.
What the hybrid tier buys
Once history is off the card, the bus is the constraint, so I judge a representation by the bytes per token it stops moving; a slower kernel is acceptable when it removes enough transfer. On one Intel Arc Pro GPU serving a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model, the two modes compare like this, beside one change that spent reclaimed memory on purpose:
Device budgets are explicit, admission does the arithmetic before it lends a page and logs it with its formula, and each context can be checkpointed. Host-memory KV streaming has public precedent and log-sum-exp merging is known mathematics; the work is a native, lifecycle-safe integration of both on a single card with eager execution, and at long contexts with a single request stock vLLM is still faster per token than either tier. The full account is in Context beyond VRAM and Judge a representation by the bytes it stops moving.
Keeping prefixes hot from the broker
Inference Fabric: for throughput serving, I saw the cache problem from the other side. The engine's prefix cache only pays off if related work arrives together, so I made the token-aware broker build shape-pure batches and send related work to the same engine. With a 32 GB card and a 16 GB card serving one backlog:
Related work goes to the same engine; a guard refuses to mix shapes inside one batch.
On that NVIDIA throughput path, offloading the cache to host memory measured slower, so I dropped offload there. See A message broker that understands tokens and From 44K to 163K tokens per second.
A paged cache beside a larger model
Capabilities model: the purpose-built CUDA decoder keeps its key-value cache in paged memory and admits requests against measured memory, so it can run beside a larger model on the same card.