Capability

Bigger models and longer context on fixed GPU memory

When GPU memory, not the workload, is the limit

C++ · SYCL · vLLM27B hybrid model, 4-bit weightsOne 32 GB Intel Arc Pro GPU
One 32 GB card · budget spent on purpose
4-bit weights, 27B bounded staging recurrent state free for concurrency
Host RAM · the history lives here
128Ktokens of history, one card
  • 13 of 13exact cells bit-identical to fully resident attention at the native port, 12 of 12 hybrid
  • 0Python files in the release payload of the memory layer
  • +36%sustained output throughput of the 4-bit build over the mixed 4- and 8-bit control, 64 clients

A model's context, or the number of long sessions that can stay alive, is capped by the memory on the cards at hand. The usual answers are bigger cards, more cards or less context.

I built an inference-memory layer for vLLM that moves a context's history into host RAM and treats the GPU as a bounded working set, with an exact tier that is bit-identical to fully resident attention. I ported it entirely to native C++ under parity gates, as a patch that applies to an unmodified vLLM. Start with Context beyond VRAM.

Where the trade sits
Mine is faster · single 8K request, time to first token
Memory layer3.68 sStock vLLM4.71 s

And under concurrency, where stock queues, inter-token latency is lower.

Stock still decodes faster · longest single request, 128K inter-token latency
Exact tier335 msHybrid tier190 msStock vLLM54 ms

Staging history across PCIe is still visible at the longest contexts; the hybrid tier is how I halved it.

Which regime a workload is in decides whether the layer pays; the boundary is measured, not promised.

I fit a 27-billion-parameter model on a single 32 GB Intel card through a published, measured quantization. See A 27B model on one 32 GB Intel card.

One 32 GB card Full precision: overflows the card 4-bit weights, with room left for context and concurrency

The outline is the card; the shapes are schematic, not to scale.

RecipeW4A16, group size 32BF16 scales, rounded straight from the original weights, no calibration data
Cost, stated3.33% higher perplexitythan full precision on an eight-document diagnostic set
Release check433 fileschecked per checkpoint, on the exact upload copy
Four arms measured on the same card, crossed over across two cards, so each gain is attributable to what changed.

I also patched a vendor's kernel stack on brand-new silicon with outputs unchanged. See Bringing a serving stack up on new silicon.

Cold 32K-token prefill
Before21.54 sAfter17.23 s
Recurrent-layer prefill, per chunk
Before53.8 msAfter27.4 ms
Kept and dropped

Kept: a one-entry GEMM patch for the Xe2 prefill shape, and exact arithmetic checked across 48 special-value cases with no speed gain. Dropped: a 64-key attention tile, slower at longer contexts, and a neutral rescale skip.

Every patch went through an identical-output gate inside the vendor's own toolchain.

Exactness is a property I want in the system.