Corey James- 12 of 12bit-identity gates across batch compositions
- 2,345of 2,345 shuffled compositions agree on the top result
- 484of 484 ranked lists match an independent implementation
Under continuous batching, the same request can return a different answer depending on what else is in the batch. For most workloads that is noise. For a model whose job is to tell an agent which tool to call, it is a correctness bug: the ranking must not depend on strangers. I set a gate that top results must be stable across batch compositions, then went further and built the decoder so that its output bits are identical whatever else is in the batch.
The challenge
GPU reductions are fast because they are parallel, and parallel floating-point sums are order-dependent. Different batch sizes choose different tiling, different tiling changes summation order, and the last bits of a logit shift. Here, two candidate tools separated by a hair could swap places depending on the batch, and an agent would call a different tool for the same question. The easy fixes, double precision everywhere or batch size one, throw away the throughput that justified a dedicated decoder.
Two partial sums, added in one order.
Three partial sums, a different order: the last bits move.
Constraints
Determinism from structure, with no slower arithmetic and no serialised execution. The same kernels for both paths, because a serial path and a batched path that differ in code will eventually differ in bits. Throughput preserved, because the decoder exists to serve two thousand clients beside a 30B model.
My approach
I ruled that top-result stability was the behaviour that mattered and gated on it. Then I asked for the stronger property, because the stronger property makes the weaker one provable: every output row is computed by exactly one block, reductions happen in a fixed order that does not depend on batch size, and the serial and batched engines share the same row kernels so there is only one implementation to be right.
What I built
One block per output row, so a row's computation never spans a partition that could change with batch size.
Fixed-order reductions: a fixed 32-lane shuffle tree within a warp and a fixed 256-thread tree above it, regardless of how many rows are in flight.
No K-split tiling, which would introduce batch-dependent partial sums, taken as a limit on kernel tuning.
Shared row kernels between the serial and batched engines, enforced by a header invariant so the two cannot drift apart.
Pairs combine in the same order whatever else is in flight.
Gates that check bit-identity across batch compositions, and a cross-check of every ranked list against an independent implementation.
Difficult decisions
Giving up tuning headroom. Forbidding K-split tiling and fixing reduction order leaves performance on the table. I took the deterministic design and recovered throughput elsewhere.
A faster decoder that sometimes picks a different tool is slower at the job.
Results
Every gate held. The cross-check of ranked lists covered all classes, both engines and two transports.
Technologies used
CUDA · Rust · Floating-point reduction design · GPU kernel architecture
What this demonstrates
Numerics judgment. I know where GPU nondeterminism comes from, I decide which property the application needs, and I design the arithmetic so the property holds by design and is checked by the gates.