Corey James- 26%faster one-row decode than BF16, reading 8-bit history directly
- 31%less device time for the two-owner decoder than two one-row calls
- 256registers per work-item; every candidate fits with no spills
- 2 KBinteger query cache held in registers
I can work at the kernel level on hardware the ecosystem has not caught up with: design the operator, fit it into the register file, prove it exact or bounded, and patch the vendor stack where it falls short.
ActiveKV runs on an Intel Arc Pro GPU, a new architecture whose open kernel stacks had gaps. I wrote two kinds of SYCL: my own attention operators for the compressed history tier and the merge that joins the tiers, built on Intel's open SYCL attention sources, and patches to the vendor's kernels, credited as such. Every figure below is on one Intel Arc Pro GPU serving a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model.
Attention that reads 8-bit history
The compressed-tier operators start from the vendor's SYCL flash-attention sources (CUTLASS for SYCL) and read 8-bit history without expanding it. I shaped the data format around the kernel that reads it, choosing each scale so it can leave the inner loop: keys carry a separable scale, a per-channel factor times a per-token factor, and values carry one scale per channel per page.
Orange: the per-token factor, applied after the dot. Values carry one scale per channel per page; PV accumulates on raw codes in half precision and is rescaled at each page boundary.
Each subgroup handles 8 query rows over 32-key tiles at head size 256, holding the integer query cache, one FP32 output accumulator and a half-precision probability plane in registers, with no spills and no shared local memory. Every candidate must fit before I time it. A two-by-two experiment decided which products run on integers.
Exact tricks and a two-owner decoder
I keep a kernel optimisation only if it is exact or passes the fixed quality gate. Two exact tricks made the cut: an eight-row vote skips the online-softmax rescale only when every row's factor is exactly 1.0, and a running reference rounded up to an integer in base 2 turns every rescale into a power of two. A two-owner decode operator serves two requests with the same page count in one launch, each with its own metadata address and its own merge accumulator.
All eight rows at exactly 1.0: the multiply is skipped.
One row differs: every row rescales, by a power of two.
Rejected on measurement: a 64-key tile (20 percent slower above 1K tokens), a rescale skip that changed nothing, centred keys, a shared-memory query cache and a persistent integer-query cache.
A merge that joins the tiers
The merge kernel runs one work-group per token and head and one work-item per channel, and folds the partials from each tier into one FP32 accumulator. The vendor's paged-decode kernel had no log-sum-exp output, so I added one, initialised to negative infinity so an unwritten row is a valid empty partial.
Patching the vendor stacks
The vendor patches build from the vendor's own sources with the oneAPI compiler inside the vendor image, so the binaries match the ABI the serving engine loads. One new entry in oneDNN's Xe2 GEMM catalogue lifted the 4-bit prefill GEMMs on the model's K=5120 shapes, with identical outputs. In the recurrent-layer prefill library, two kernels ran on 16 work-groups, each looping over all 48 heads, and giving each chunk and head its own work-group fixed that.
The bit-for-bit comparison against stock then exposed a cross-subgroup visibility race that stock avoided only by timing. Five barriers that also publish global memory close it. Output stayed bit-identical to stock, and decode log-probabilities became repeatable from run to run.
A bitwise mismatch is a finding.
Later patches reworked the 4-bit GEMM lowering: a wider workgroup for weight reuse, integer conversion fused with scaling, one instruction that widens and broadcasts each scale, and pruning of inactive systolic columns on a hardware tile tail. Recurrent-layer temporaries became one slab, and normalised queries and keys are now shared across subgroups. A validation switch makes 4-bit GEMM reductions deterministic and leaves release builds bit-identical when it is off. I kept one GEMM rewrite that gave exact instruction-level arithmetic across 48 special-value combinations with no speed-up, because it removes work and stays exact. See Bringing a serving stack up on new silicon and Judge a representation by the bytes it stops moving.