Engineering story · ActiveKV

Judge a representation by the bytes it stops moving

Rewriting an acceptance gate that kept rejecting good compression, and what it bought at 128K tokens

4 min readActiveKVSYCL kernelsQuantisation
One decode step at 128K, exact tier · where the time goes
moving history across PCIe, about 327 msattention, about 16 ms
The gate that matters · bytes no longer moved
The same history in a compact form the GPU reads directly: half the bytes across the bus, a slower kernel, a faster step.
190 msper token at 128K, from 335 on the exact tier
  • 0.7×the per-tile speed rule I had set, withdrawn after a false rejection
  • 12 of 12bitwise cells for the hybrid route against its own reference
  • 3 of 3context sizes measured where the hybrid tier decoded faster

Once a model's history lives in host memory, every decode step is bound by how many bytes cross the bus, and the attention kernel's speed is a rounding error. My acceptance gate for compressed history formats did not know that. It measured kernel speed per tile and kept rejecting candidates that would have won. I rewrote the gate around end-to-end economics at the target context size: bytes saved per token against GPU time added.

The challenge

ActiveKV stages a context's attention history across PCIe on each step. At long contexts the copy dominates. A planning analysis for the exact tier put the transfer at many times the cost of the attention itself at 128K. A compressed format that halves the bytes should nearly halve the step. Its kernels are slower per tile than the exact ones, and a gate that compared kernel speed saw only the slowdown.

Constraints

A lossy tier needs a quality contract of its own; earlier 8-bit screens had failed their loss limits. The compressed and exact parts of a history had to combine without approximation. And the same single card, one Intel Arc Pro GPU, with no second GPU to hide the transfer behind.

My approach

Replace the per-tile rule with a whole-step economic one. For a candidate representation at the target context size: how many bytes does it stop moving per token, what does that save in transfer time, and how much GPU time do its kernels add? Accept if the net is positive at the sizes that matter.

A format that loses at 8K and wins at 128K is a format for long contexts, and the gate should say so.

What I built

A compressed history representation and a hybrid serving mode, each with its own quality contract and its own reference.

The economic gate itself, with a transfer model derived from measured bus behaviour on the card, so candidates are judged on bytes saved per token at each target size.

Difficult decisions

Overturning a rule I had set. The earlier rule, a fixed ceiling on a candidate's cost per tile against the exact tier, was mine. At 128K it produced a false rejection. I withdrew it and wrote the replacement with the physics beside it.

the rule I withdrew · kernel cost per tileexactcompactslower: rejectedthe rule that replaced it · the whole step at the target sizeexactcompactfaster: kept
moving history across the bus
attention kernel
Proportions illustrative.

Keeping the quality bar. The economic gate made it easier to accept a compressed format. It did not touch the loss limit.

Results

Single request, one Intel Arc Pro GPU, a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model:

Per-token latency · single request, ms
31K exact91.531K hybrid56.864K exact173.664K hybrid102.9

Stock vLLM remains faster per token at these single-request contexts; that comparison is on the ActiveKV page.

Technologies used

SYCL kernels · Weight and activation quantisation · PCIe transfer modelling · vLLM · Intel Arc GPUs · KV-cache engineering

What this demonstrates

First-principles systems economics over local optimisation. I will overturn my own rule when the physics says it is wrong, and I keep the quality bar in place while I do it.