Corey James- 29kernels in 13 families, with exact, inverted-file and lattice search
- 59kernel and header files in the analytics engine, tests included
- ~500,000vectors a second for two-means clustering, rewritten without atomics
- 2,740MiB, the batched decoder's footprint beside the serving model
Cluster-health kernels with no library to lean on
In the Inference Fabric I needed health statistics for every embedding cluster computed on the GPU next to vector search, over millions of vectors on consumer cards, and no library offered them, so I had the kernels written.
Every kernel takes all clusters in one launch. Linear-time statistics run one block per cluster with a grid-stride loop and block reductions. Pairwise statistics run one block per pair of tiles; the dimension loop sits outside the pair loop so every thread reaches every barrier.
Memory is bounded by tile size, whatever the cluster size; atomics survive only as one sparse add per tile.
Rewriting a kernel whose memory traffic was wrong
The centroid kernel shows the style. Its first version accumulated the centroid with atomics and re-read every vector for threshold counts. I had it rebuilt in three passes, and it now scales linearly from 1,000 vectors.
Two-means clustering was rewritten the same way, dimension-parallel. Statistics that need a median of all pairwise distances use a 256-bin shared-memory histogram filled during the tile loop.
Search that never builds the score matrix
Search uses one block per query with a top-k buffer in shared memory, and a shared two-pass top-k engine merges partial results for every search family. Race-freedom and cross-block tests run beside every metric family and search executor.
Every kernel has a CPU reference and an equivalence test, so a fast kernel is never the only definition of the answer.
A decoder whose bits never depend on the batch
In Capabilities model, the tool-routing model had to share a GPU with the serving model and return the same answer whatever else was in the batch, so I had a decoding engine written for it in CUDA, across decoder, grammar, batched, prefill and NVFP4 kernels.
On this hardware cuBLAS defaulted f32 GEMM to TF32 and changed reduction order. I chose determinism from structure instead: one block per output row, a fixed reduction order and no K-split tiling. The decode loop runs as a device graph with a device-side conditional. Before any NVFP4 kernel existed I had the packed layout identified by correlating candidate decodings against the same weights in the fp32 checkpoint.
about 518 µs removed from every token on a quiet Blackwell GPU
about 5 percent slower at batch 1
See Bit-identical across batch compositions and A router beside a production model.
Where my CUDA ends
ActiveKV started from a problem on a Blackwell GPU: vLLM's KV offload did not handle hybrid-attention models, so I patched vLLM's KV-offloading connector for them, including a change to FlashInfer's recurrent-prefill launcher.