Engineering story · Inference Fabric

Removing the sampling caps

From kernels that looked at fifty members and twelve hundred pairs to exact statistics over a hundred thousand members, by redesigning every kernel into one of three scaling classes

CUDA · RustThree scaling classesNative Blackwell binaries
Before · a cluster scored from a sample After · every member, every pair
Was a sample of 50
100,000members per cluster, all of them scored
  • 1,225pairs the old pairwise statistics were capped at
  • 2,000member limit for Gram-bounded kernels, refused above it
  • 6accumulator types in the shared tile library
  • 20legacy kernels kept, with sixteen tests against the references

The first generation of my cluster-metric kernels cheated, and I knew it. A cluster of a hundred thousand embeddings was scored from a sample of fifty members. The two-means split and the silhouette ran on fifty-member subsets. This is the story of taking every cap out.

The numbers came back fast and they were estimates of the thing I wanted rather than the thing itself.

Why the caps were there

The early kernels were written one metric at a time, each with its own loop over members, each accumulating into global memory with atomics. At fifty members, contention on those atomics was invisible. At a hundred thousand it would have dominated, and a pairwise kernel over a hundred thousand members has five billion pairs. Sampling was the quick way to keep both problems out of sight.

Three classes instead of one

The rewrite sorted every kernel into one of three scaling classes with a fixed shape each.

Streaming · linearAny cluster size
one block per cluster, grid-stride over members, block reductions only
Tiled pairwise · quadraticEvery pair, once, in a fixed order
16-by-64 tiles, one block per tile pair, registers until the tile ends, one atomic per tile
Gram-boundedMemory grows with the squarean explicit member limit, with a documented refusal above it

The tile loop runs with the dimension on the outside and the pair on the inside, so every thread reaches every barrier, and each kernel declares which accumulator it carries, so the loop stays the same.

One tile · each cell a pair
cosine
distance
redundancy counts
information-density counts
concept breadth
Gaussian entanglement sum

Removing atomics where they contended

Two kernels were rewritten with no atomics at all. The centroid kernel became three passes: a dimension-owning block reduction that writes each cluster's centroid once, a single sweep over the members that computes distances and norms into Welford accumulators, and a scan of the distance buffer for the sigma-threshold counts. The centroid it writes is reused by every downstream kernel. The two-means kernel keeps its assignments in a byte per member and its centroid update dimension-parallel, and runs at about half a million members per second, linearly.

The numbers

Centroid family · milliseconds
1,000 members0.4510,0004.52100,00046.67

521 ms at 100,000 members before the rewrite, eleven times slower.

Neighbour relevance · milliseconds
1,000 members2.2210,00022.48100,000231.82

Its own and five neighbouring centroids cached in about 24 KB of shared memory.

The silhouette kernel runs in under 1.2 milliseconds on the cluster sizes it sees.

What was kept

The legacy kernels are still in the tree, with their tests against the CPU references, so a new kernel can be compared with the generation it replaced and a regression has a baseline to be measured against.

What this demonstrates

A performance rewrite driven by a correctness goal. The caps were the shortcut, and taking them out meant choosing a scaling class for every statistic, bounding memory by design, and proving each kernel against a definition rather than against its faster self.