Corey James- +36%sustained output throughput over the mixed 4-bit and 8-bit control, 64 clients
- 3.33%higher perplexity than full precision on an 8-document diagnostic set
- 1,798tokens per second total at 32 clients, 1,536-token prompts, prefix caching on
The base model, a 27-billion-parameter hybrid recurrent-attention open-weight model, does not fit in BF16 on a 32 GB Intel Arc Pro GPU, and I needed it to, with room left for context and concurrency. I built a 4-bit-weight quantization that fits it, ran a four-arm experiment to attribute exactly where the speed came from, and published the model with its card on Hugging Face. The model card states the trade-off in plain numbers.
The challenge
Fitting a large model on a smaller card is routine on NVIDIA hardware with mature tooling. On a new Intel architecture, with a stock XPU build of vLLM and no forked kernels, it was not. And a quantization you intend to publish has to be provable: which source weights, which revision, which tensors changed and which did not, measured on your own diagnostics.
Constraints
Stock runtime: the model had to run on the unmodified Intel XPU vLLM build, so the format had to be one the runtime already understood. Provenance: source pinned to a revision with every shard hash checked, every quantized module listed in a manifest. Attribution: a speed gain can come from smaller weights, from different kernels, or from the memory freed for cache, and I wanted to know which.
My approach
A four-arm experiment on the same card: the full-precision reference, a mixed 4-bit and 8-bit control, a partial 4-bit hybrid, and an all-4-bit hybrid. Each arm was measured on the same card, with the order crossed over across two cards so no arm benefited from a particular device or warm state. The gain of each arm is then attributable to what changed between it and its neighbour.
What I built
A weight-only W4A16 recipe: 4-bit symmetric weights, group size 32, BF16 scales, rounded straight from the hash-checked original weights with no calibration data.
Per-module manifests and hash-replayed measurements, so any figure traces to the artefact it was measured on.
Release validation on the exact upload copy, 433 files checked per checkpoint.
A model card that states the method, the trade-off and the measurement conditions.
Difficult decisions
Choosing the all-4-bit arm. It cost some perplexity, it bought throughput, and it moved fewer bytes per generated token.
I took the throughput and stated the cost.
Skipping calibration. Calibrated quantization can be closer to the original. Round-to-nearest from hash-checked weights is simpler, reproducible from the source alone, and leaves nothing to a calibration set that readers cannot inspect. I chose provenance.
Making no "best" claim. The card describes what this model is and what it costs, and leaves comparison to the reader.
Results
On one Intel Arc Pro GPU, the all-4-bit model against the mixed control:
The model my ActiveKV work serves is a same-recipe build of a different checkpoint.
Technologies used
Weight-only W4A16 quantization · compressed-tensors format · Intel XPU vLLM · PyTorch XPU · Intel Arc GPUs · Hugging Face Hub publication · Hybrid recurrent-attention 27B-class models
What this demonstrates
Model-deployment engineering on non-NVIDIA hardware, an experimental design that separates kernel effects from capacity effects, and publication that tells the reader exactly what they are getting.
The base model is a third party's open-weight release; the runtime is Intel's.