Corey James- 48special-value combinations checked in one exact, flat rewrite
- 2attention candidates rejected and kept as negatives
- 5memory fences added for a synchronisation the original lacked
The Intel Arc Pro GPU is a new architecture, and the open kernel stacks that serve large models on it had gaps when I started: a GEMM path that chose the wrong tile for prefill, a recurrent-layer prefill with poor occupancy and a missing synchronisation, a missing attention statistic my memory layer needs, and run-to-run variance in some GEMM shapes. I patched the vendor's own sources inside the vendor's toolchain, with an identical-output gate on every change. Outputs stayed unchanged on the gated path.
The challenge
Serving on brand-new hardware means working above a kernel stack that is still catching up with the silicon. The vendor's libraries were open, which made patching possible, and a patch to a GEMM library or an attention kernel that changes a single bit of output invalidates every parity result above it. I needed speed where the stack was slow, exactness where the stack was approximate, and a way to prove both for every change.
Constraints
The vendor's sources and the vendor's toolchain, so the ABI matched what the serving engine loaded. Identical outputs against the unpatched path on the quiet benchmark host. Negative results kept. And honest attribution: these are patches and extensions to Intel's open stacks.
My approach
One change, one patch, one gate. Each candidate was a self-contained patch with an identical-output check and a whole-workload measurement. I kept real systems improvements, fewer instructions, fewer allocations, exact arithmetic, even when the whole-workload metric was flat, because correctness and stability improvements are worth keeping on their own. I rejected speed-ups that changed outputs or won only on a narrow shape.
What I built
A one-entry GEMM catalogue patch in oneDNN for the Xe2 prefill shape, which is where the prefill gain came from.
A recurrent-layer prefill fix: occupancy, output split, and memory fences for a synchronisation the original lacked.
The missing attention statistic, added to the paged attention path.
A validation determinism switch that pins the variable GEMM shapes for parity runs.
Post-port lowering patches for the 4-bit-weight model path, and scratch-slab and query/key reuse patches in the recurrent layer, which cut instructions and allocations.
Difficult decisions
Rejecting attention candidates that did not earn their place, and keeping them as negatives.
Accepting a flat change. One kernel rewrite produced exact instruction-set arithmetic with no whole-workload gain.
Exactness is a property I want in the product.
Results
On one Intel Arc Pro GPU serving a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model:
The patches are delivered with ActiveKV as patches to the vendor stacks.
Technologies used
SYCL and DPC++ · oneDNN · CUTLASS and CuTe for SYCL · Intel's vLLM XPU kernels · Intel Arc GPUs · Bit-identical validation
What this demonstrates
Low-level GPU engineering on hardware the ecosystem has not caught up with yet, with honest attribution to the vendor base I built on, and the discipline to keep every output bit where it was.
The kernel stacks are Intel's; the patches are mine.