Corey James- 8.4 GBmoved across PCIe per generated token at 128K on the exact path
- 0.0027worst merged error, against a bound of 0.005
- 48real samples the loss bound is checked on
- 512own-task calibration samples for the router's FP8 and NVFP4 builds
A lossless tier first
ActiveKV keeps attention history in host memory, so every decode step pays to move that history across PCIe. A smaller history buys time on every token, but only if the GPU can attend to the small form directly. I tried lossless compression first and let the bus arithmetic decide when a lossy format was worth its loss bound.
13 bits per value, an 18.7 percent saving, with a raw fallback when a plane needs more than the 32-entry table.
It matched BF16 serving speed, and the bus arithmetic said 19 percent was too small to matter at long context.
An 8-bit page format shaped for its kernel
The current tier is an 8-bit page format I designed around the attention kernel that reads it. Keys get a separable scale. Per channel, the scale is the square root of that channel's RMS over the page, normalised so the channel scales multiply to one and clamped to 2^±14; it absorbs half of each channel's dynamic range. Per token and head, a second scale covers the rest, and codes are symmetric in −127 to 127. Values get one scale per channel per page.
Conversion runs on background threads only for pages fully written in every layer. Until a page is published it is served exact, so conversion never stalls decode. I judge a quantized representation by the bytes it stops moving at the target workload.
On a transfer-bound path a slower kernel that moves fewer bytes wins.
Speed earned under a fixed loss bound
The format answers to a fixed loss bound against its own higher-precision reference on real samples. The first 8-bit screens had failed theirs, and the tier kept its bound after its speed gate got easier to pass. Getting there took a sequence of matched kernel experiments.
The hybrid tier serves a 4-bit-weight build of a 27-billion-parameter hybrid recurrent-attention open-weight model on one Intel Arc Pro GPU. The 4-bit weights leave room on a 32 GB card for working memory and concurrency. See Judge a representation by the bytes it stops moving.
FP8 over INT4, by measurement
Inference Fabric served a 7-billion-parameter dense model on NVIDIA TensorRT-LLM, and the question was which precision gives the most throughput per card. I measured both on the two NVIDIA Blackwell GPUs.
See From 44K to 163K tokens per second.
Files that reproduce byte for byte
Capabilities model needed a small quantized model whose files I could trust. I quantized the fine-tuned 0.6-billion-parameter model to FP8 and NVFP4 with NVIDIA's Model Optimizer in the vendor's container, calibrating on samples from the model's own task. A quantized artefact must reproduce byte for byte, so every published number traces to one exact file.
A published 4-bit build of a 27B model
The related published work is a 4-bit build of a 27-billion-parameter hybrid recurrent-attention open-weight model for the Intel Arc Pro GPU. I chose round-to-nearest with no calibration, which anyone can reproduce from the source weights, and I put its cost on the model card.
- Schemeweight-only W4A16
- Weights4-bit symmetric
- Group size32
- ScalesBF16
- Calibrationnone · round-to-nearest
The throughput gain is measured against a mixed 4-bit and 8-bit control on the same card, and the perplexity cost against full precision on my diagnostic set. A four-arm experiment crossed over across two cards separated the gain from smaller weights from the gain from freed memory. See A 27B model on one 32 GB Intel card. The underlying quantization math is standard; my work is formats shaped for the kernels that read them, the tiers and kernels around those formats, and the measurements of what each choice buys.