Engineering story · Inference Fabric

Every metric gets a CPU twin

Thirty-eight GPU cluster metrics defined by a nine-thousand-line CPU reference library, and a three-way consistency check between kernels, references and documentation

Rust references · CUDA kernels13 reference modulesNative Blackwell binaries
Three copies of every metric · each checked against the other two
Every group passes the matrix
38metrics on the GPU, each checked against its reference
  • 9,000lines of plain Rust that define every metric
  • 32Rust integration tests, plus fifteen CUDA test files
  • 3spectral metrics that exist only as the CPU reference

A GPU kernel that computes a statistic is a poor definition of that statistic. It is tuned for throughput and it reorders its reductions. So every metric my CUDA engine computes exists twice: once as a kernel and once as plain Rust on the CPU, and the CPU version is the definition. The reference library is about 9,000 lines across thirteen modules, cohesion, diversity, substructure, spatial, dual-space, quantum, spectral, sigma, composites, exact search and top-k among them. The kernels are checked against it, and the documentation is checked against both.

What the twin is for

The reference is written for clarity, not speed, so a reader can confirm the formula by inspection. It is also where the edge cases are decided, and the kernel must match the reference on them.

When a kernel is wrong, it is wrong at speed.

The kernel
tiled, streamed, reordered reductions, fast-math
The reference · one plain loopstates the edge cases: a singleton cluster's silhouette; a histogram whose maximum equals its minimum; an empty inverted-file cell keeps its centroid

Three metrics in the spectral family live only in the reference. They need the eigenvalues of a Gaussian Gram matrix, which caps them at 2,000 members, and until the kernel exists the CPU version is the metric.

The tolerance

Dot products, cosines
1 in 10,000 relative, floor 1 in 100,000
Squared distances
1 in 10,000 absolute
Search results
zero
Tighter tolerances fail for no reason on 1,024-dimensional vectors: the two sides sum in different orders, and fast-math approximates transcendentals and divisions. Band widths are illustrative.

The triangle

A kernel, its reference and its lineage file can each drift from the other two. The lineage file for every kernel group records the formula, the kernel, the Rust wrapper, the destination column in PostgreSQL, the edge-case conventions and the measured timings. A consistency matrix checks all three against each other for every group: GPU against CPU, CPU against the documented formula, documented formula against the GPU. Every group passes. When the check was first run it found the gaps it was built to find, including metric identifiers that meant different things in the database layer and in the kernels.

Tests as the proof

Rust integration tests
every metric family, exact search, top-k across layouts, the lattice engine, fused search and health, the lease adapterCUDA test files
block Welford, reductions, the tile loader, the pair-index decoder, multi-tile processing
Older kernel generations keep their own tests, so a new kernel can be compared with what it replaced.

What this demonstrates

Numerical GPU code held to a written definition rather than to its own output. The reference library is the contract, the tolerance is derived from the arithmetic, and the consistency check keeps code, reference and documentation from drifting apart.