Corey James- 400text projection modules quantized to 4-bit
- 799tensors left bit-identical to the source
- 433files checked on the uploaded copy
A 4-bit checkpoint for one Intel card
ActiveKV needed a 27-billion-parameter model on one 32 GB Intel Arc Pro GPU, and the hybrid recurrent-attention open-weight model I chose does not fit on it in BF16. I decided to quantize it myself on Intel's stock XPU vLLM build, measure the trade-off, and publish the result on the Hugging Face Hub so anyone with an Intel Arc Pro GPU could run it.
The recipe is weight-only W4A16 on the text projection modules, with every other tensor bit-identical to the source. I pinned the source to a revision and checked every shard hash, and a per-module manifest lists what was quantized. I skipped calibration, so the checkpoint can be reproduced from the source weights alone.
At 64 clients, about 294 to 297 output tokens per second. Perplexity is against full precision on an 8-document diagnostic set.
The model ActiveKV serves in its own benchmarks is a same-recipe build of a different checkpoint. The full story is A 27B model on one 32 GB Intel card.
A card that states the cost
I published it under the Apache-2.0 licence with the base model declared and the quantized relationship set in the card metadata. The model card carries the method, the exact tested environment (one Intel Arc Pro GPU, Intel's XPU vLLM container), a run command, the measured trade-off and the places the model was not tested. It states cost and gain and leaves comparison with other quantizations to the reader.
I ran the release checks on the bytes that were uploaded, because a copy step can change them.
- Loadspassed
- Chat with reasoning onpassed
- Chat with reasoning offpassed
- A parsed tool callpassed
- Every file checkedpassed
A community discussion opened on the repository within a day. I can take a model to a new accelerator, attribute the speed I gained to its causes, and publish exactly what a user gets.
Models a person approves
Helix includes a media studio where agents render images and video on local GPUs, and those models come from the Hub. Downloads are large and licence-bound, so I decided an agent may propose a download and only a person may approve it.
Model approval happens only at the local console, and no remote call can approve a download. A background reconciler finds files added to model folders by hand and reports their check status, so a host with no models reports zero. I publish one model on the Hub and consume others through it.