Corey James- 13 of 13exact cells bit-identical to fully resident attention at the port, 12 of 12 hybrid
- 1 of 867former Python dispatch sites still in Python, kept by a cited framework contract
- 12–22%lower time to first token than stock vLLM, single request, every context length
- 20%faster cold 32K-token prefill from the Xe2 kernel patches
Every serving engine I have worked with assumes a model's working memory lives on the GPU. That assumption is why a 32 GB card decides how much context a model can hold. I decided to remove it. ActiveKV is an inference-memory layer I architected for vLLM: the conversation's attention history lives in pinned host RAM, and the GPU keeps a bounded, budgeted working set. On one Intel Arc Pro GPU, a 27-billion-parameter model serves 128,000-token histories. On a single 8,000-token request it beats stock vLLM on time to first token, inter-token latency and throughput. The runtime is C++ throughout, ported from a Python prototype without changing an output bit, and it is delivered as compiled binaries plus a patch to the user's own vLLM.
The idea
Dense-attention history is written to host memory as the authoritative copy. The GPU holds the slices each decode step needs, staged across the bus under an explicit budget. Recurrent-layer state stays on the device, because it is small and it is touched every step. That one inversion lets a single workstation card serve long documents, whole codebases and agents that have been working for hours.
Context length should be a host-memory budget, and it should stop being a hardware limit.
Two precision tiers serve that history: the exact tier, and a hybrid tier that keeps recent context exact and holds older history in a compact form the GPU reads directly; the two parts are combined with exact attention arithmetic, so long contexts decode at about half the per-token latency of the exact tier with no approximation in the merge.
What I built
The host-authoritative KV tier is pinned host memory holding every layer's history, a per-layer staging working set on the GPU with a stated budget, and fence-safe asynchronous staging with an explicit lifecycle, so a buffer is never reused while a copy or a kernel still touches it.
The two precision tiers have separate contracts. The exact BF16 path is held to bit-identical output. The hybrid path is judged on bytes saved per token against GPU time added at the target context length. That gate is economic: a slower kernel that moves fewer bytes wins on a transfer-bound path. I had to overturn my own earlier per-tile rule to get there, which is the story in Judge a representation by the bytes it stops moving.
Device memory budgets, admission and refusal are explicit, and nothing stalls.
The engine is native C++, bound into vLLM and PyTorch through pybind11 and ATen, with a native launcher. The release payload contains no Python of mine; the one remaining Python dispatch is justified line by line against the framework contract that requires it. See Rewriting an engine extension without changing a token.
I patched the vendor's open kernel stacks on Intel Xe2 where they had gaps: GEMM tile selection for prefill, occupancy and a missing synchronisation in recurrent-layer prefill, and the attention statistic the exact cross-tier merge needs. Each patch was gated on identical outputs, and recurrent prefill halved per chunk. See Bringing a serving stack up on new silicon.
Instrumentation follows each token from scheduler to device to client, at selectable levels, with its own overhead measured so that instrumented runs are never quoted as product speed. The day the profiler turned out to be causing the serialisation it reported is The observer that changed the answer.
The release process produces binaries and a patch with build receipts, keeps diagnostic builds separate, and runs every performance change as an A/B on a quiet host through an intake review that rejects gains tuned to the test fixture.
Results
Measured on one Intel Arc Pro GPU serving a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model, quiet host, instrumentation off.
8K prompts at 32 requests and 31K prompts at 8. Stock queues the requests it cannot hold; ActiveKV serves them, with aggregate throughput close to stock.
Histories from 32,000 to 128,000 tokens are served on the one card. At the longest single-request contexts stock vLLM still decodes faster per token, because every step pays to stage history across the bus. That is the trade ActiveKV makes for capacity, and the hybrid tier halves the cost. The release passed twelve of twelve on both routes.
What it demonstrates
Deep integration into a third-party engine's scheduler, worker, attention backend and memory accounting. Lifecycle-safe asynchronous host-to-device staging. Exact attention merging across precision tiers. A lossy compression tier with its own kernels and quality contract. A full Python-to-C++ port under bitwise-parity gates. Kernel tuning on a new accelerator architecture. And the discipline to build instrumentation that does not change what it measures, and to retract the one finding where it did.
Stories
Context beyond VRAM · Rewriting an engine extension without changing a token · Proving a rewrite is the same system · Judge a representation by the bytes it stops moving · The observer that changed the answer · Bringing a serving stack up on new silicon · A 27B model on one 32 GB Intel card · Half the bytes, exactly merged · Staging without a stall
Technologies
C++ · SYCL and oneAPI · vLLM internals · PyTorch XPU and ATen · pybind11 and the CPython C API · oneDNN · GPU memory management · KV-cache engineering · Intel Arc GPUs · Bitwise parity validation · Instrumented performance methodology