Corey James- 147 MiBrecurrent-layer state per context, fixed whatever the length
- 2 GiBof pinned host memory per 32K-token history
- 16dense layers sharing one device arena
- 0CUDA allocations after the first second of a trace, down from about 4,200
Inverting who owns memory
ActiveKV began with a constraint I did not accept. Stock vLLM keeps a request's whole key-value cache on the GPU, so on a 32 GB Intel Arc Pro GPU the card decides how much context a model can hold. I inverted who owns memory and gave the device a stated budget, so the operator chooses the trade-off.
I give GPU memory owners, budgets and lifetimes, on two vendors' hardware.
Memory is a design decision, not a default.
Pages held by right, borrowed or free
The device side is one arena shared by every dense layer: staging pages for 32K-token tiles plus a pool of pages. Each context has a target it holds by right, a burst it may borrow while pages are free, and a floor it needs to run. On every step the engine keeps the newest pages and an attention-sink page hot, and pages that turn hot below the current frontier are copied in once.
Reclamation runs least-recently-active first, in a fixed order, and every step of it is counted.
A host pool that explains its own size
Startup computes how many histories fit on the host and prints the formula, so a refusal or a reduced concurrency limit explains itself.
Each group is one history: a segment per dense layer of context length × 4,096 bytes. The leftover segments cannot hold a history, so concurrency is lowered and vLLM queues the rest, or the configuration is refused with the setting to change.
The engine refuses an automatic device pool size outright. I allow explicit pool sizes only, because an automatic size hides the decision the operator is making.
Staging as a lease
Staging is a lease with a lifecycle, and its stage generation only increases. Copies run on their own H2D and D2H queues, and attention waits on the copy event on the device. Only the changed tail rows are written back, after attention signals.
A buffer is released only on signalled fences, after waits on every copy and kernel that touched it, and an unknown fence outcome quarantines it. Copy tickets keep a lease on the host pages, so a page cannot be recycled while DMA reads it.
Spending the memory it frees
Moving history to the host frees device memory, and I spend it on purpose. One change retains each context's immutable history on the device and materialises it with a device-to-device copy that overlaps the incoming suffix. A second change keeps that history in revocable headroom, device memory the engine profiled as free and can take back under pressure. I spent about 8 GB of it to roughly halve long-context latency under concurrency. The full design is Context beyond VRAM.
Inter-token latency in milliseconds; both latency panels share one scale. One Intel Arc Pro GPU, a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention model, instrumentation off.
ActiveKV runs on one card in eager mode, and at the longest contexts the cost of staging across PCIe still makes it slower per token than stock vLLM.
Nothing allocated after warm-up
Inference Fabric had two memory problems. The throughput runtime around TensorRT-LLM had to hold steady for hours, so I ruled that it allocates everything at warm-up. The runtime rejects synchronisation on the compute stream and forbids CUDA-graph capture on its own streams.
The Vector Shard Manager had to hand stored vectors to hand-written CUDA kernels. I put a type-only interface crate between storage and CUDA, and a budgeted residency manager decides what stays on the GPU and can explain each decision.