vLLM

Moving a request's history off the card

ActiveKV began with a limit I kept hitting in vLLM: it keeps a model's whole key-value cache for a request in GPU memory, and its offload is a prefix cache that reuses prompts it has seen. Once a request's history outgrew the card, the request could not continue. I treated prefix reuse and active-context tiering as separate problems from the start.

Prefix cache · reused prompts

Tiering · one request's history

pinned host memory

bounded working set on the card

I decided to move the history to host memory and keep it inside vLLM, at its own extension points, delivered as compiled binaries plus a patch to the user's own vLLM, so users keep their own engine. Attention history lives in pinned host memory and the card keeps a bounded working set. On one Intel Arc Pro GPU it serves a 27B hybrid model with long histories, and on a single 8,000-token request it beats stock vLLM on the same card on every metric I track.

At the longest contexts with a single request, stock vLLM is still faster per token. See Context beyond VRAM.

From Python inside vLLM to native C++

The runtime then moved from Python inside vLLM to native C++. Exact mode requires output bit-identical to fully resident attention, and the port kept it bit-identical in every parity cell; see Rewriting an engine extension without changing a token and Proving a rewrite is the same system.

Python inside vLLM

Native C++

A 27B model on a stock build

The Capabilities model work needed a 27B model on the same card. I quantized a 27-billion-parameter hybrid recurrent-attention open-weight model to 4-bit weights in a format Intel's unmodified XPU build of vLLM already serves, so no kernel fork was needed. Against a mixed 4-bit/8-bit control on the same card it gave more sustained throughput for a small rise in perplexity over full precision. See A 27B model on one 32 GB Intel card.

The model's evaluation scaffold also ran vLLM with guided decoding as a comparison arm, and I corrected an early finding that concurrency added nothing: the client was misusing continuous batching.

A flat throughput curve under concurrency now sends me to the client first.

The published path

27B hybrid open-weight model

4-bit weights, a format the stock build serves

Intel's unmodified XPU build of vLLM

One card

Not needed

Custom kernels

✕Kernel fork

Local models for agents

Agent Swarm runs agents on local open-source models served by vLLM through an endpoint pool. One of those agents edited, tested and proposed a change with no access to the repository credential; see A coding agent that never holds the keys.

vLLM endpoint pool

Agent on a local model

Edits and tests

Proposes a change

The repository credential

Agent

✕No access

I can change how a large serving engine manages memory from inside its extension points, keep its output exact, and measure the result against the stock engine on the same hardware. vLLM is the open-source project's; ActiveKV, the quantization and the integration code are mine.