Corey James- 4patches to Intel's own kernel sources, each with identical outputs
- ~36%more sustained throughput, all-4-bit against a mixed 4-bit and 8-bit control
- 3.33%higher perplexity paid for that throughput, eight-document diagnostic set
Long contexts on one 32 GB card
ActiveKV: I wanted a 27B hybrid model to serve long contexts on one 32 GB card. On a 32 GB Intel Arc Pro GPU the key-value cache was the wall, and I chose to make that card the target and fix the stack where it fell short. ActiveKV keeps attention history in pinned host memory and gives the card a bounded working set, so a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model serves histories from 32K tokens upward on a single card.
On the release build, an 8,000-token single request against stock vLLM on the same card:
The work is single-card with eager execution, and at long contexts with a single request stock vLLM is still faster per token. The details are in Bringing a serving stack up on new silicon and Context beyond VRAM.
Patching Intel's kernels without changing an output
The card is a new architecture (Xe2), and the open kernel stacks had gaps when I started. I kept the stock vendor runtime wherever it worked and patched Intel's own open sources inside Intel's image, for ABI match: a oneDNN kernel-catalogue entry so large prefill GEMMs pick a better tile; occupancy, output-split and memory-fence fixes in the recurrent-layer prefill; a paged-decode log-sum-exp output, so attention computed across memory tiers merges exactly; and a switch that makes int4 GEMM reductions deterministic for parity runs.
I also measured two attention-kernel candidates, a 64-key tile and a rescale-skip, and rejected both; that negative result stays in the plan with them.
Every kernel patch had to produce identical outputs before I took any speed from it.
A published 4-bit build of a 27B model
Capabilities model: the 27-billion-parameter hybrid recurrent-attention open-weight model is too large for the card in BF16, and I wanted it whole on one 32 GB card with room left for context and concurrency. I built a weight-only W4A16 quantization of its text projection modules with symmetric 4-bit weights, in a format Intel's stock XPU build of vLLM already reads with its existing wNa16 kernel.
I chose provenance over calibration, so anyone can reproduce the build from the pinned source alone. A four-arm experiment, run in crossover order across two of these cards, attributed the gain: the all-4-bit build won on sustained throughput against a mixed 4-bit and 8-bit control, for a small perplexity cost. I published it. See A 27B model on one 32 GB Intel card.
Measuring a card without distorting it
A trace suggested host-to-device copies were not overlapping with compute on the card. A controlled experiment showed the timing events themselves were serialising the copy engine, and I retracted the conclusion.
The events serialised the copy engine, so the trace showed no overlap.
Measured another way, the overlap the trace denied was there.
That is The observer that changed the answer. Being able to say exactly where each gain came from, and where a measurement was wrong, is the point of the whole programme.
A local model endpoint for the voice engine
Helix: the voice engine needed a local language model that answers fast enough for conversation. An Intel Arc Pro GPU serves the 27-billion-parameter hybrid model with prefix caching as one of the engine's local endpoints, reached over an OpenAI-compatible streaming interface.