Technology · Discipline

Cluster-health metrics on the GPU

Per-cluster metrics, each answering a question a corpus manager asks, computed across thousands of clusters in fused GPU launches and defined by CPU reference code.

CUDA engineInference FabricCPU reference for every metric
Clusters of every size · one batch
Centroid first every other family reads it
38metrics for every cluster
  • 6families of metrics, each answering one kind of question about a cluster
  • ~12fused GPU launches for a batch of thousands of clusters
  • 8operational metrics about a cluster's life in the database, beside the geometry
  • 5composite scores, weighted from a table in the database

What the engine measures

My distillation research keeps millions of embeddings in clusters, and the corpus is managed by what the clusters measure: a cluster is split, merged or left alone on the strength of its scores. I built a CUDA engine that computes every metric across every cluster at once, under the Inference Fabric on the same leased GPU memory as the search kernels.

CentroidHow do members sit around the centre?
PairwiseHow do members relate to each other?
DiversityHow much ground does it cover?
SubstructureDoes it want to be two?
NeighbourhoodWhere does it sit among the rest?
GaussianWhat does it look like as a field?
Members around a centroid, drawn for illustration.

One block per cluster, one block per tile pair

Clusters of any size are processed together through compressed-row offsets. Streaming metrics get one block per cluster; quadratic metrics get one block per tile pair.

The centroid family is the streaming foundation. One three-pass kernel writes each cluster's centroid once and produces cohesion, inertia, vector spread, density skew, embedding quality, novelty against the global centroid, the outlier ratio beyond two standard deviations and the core-periphery ratio. The other families read that centroid, and the rewrite that removed its atomics is told in Removing the sampling caps.

The pairwise family is tiled over every pair: coherence, the mean pairwise cosine rescaled to the unit interval, and compactness, one over one plus the mean pairwise distance. The diversity family extends the same tiles into entropy over a twenty-bin distance histogram, coverage, redundancy, information density and concept breadth. It needs two passes because entropy wants its range first, as told in Diversity in three kernels.

Streaming · one block per cluster
Quadratic · one block per tile pair

Splitting, neighbours and fields

The substructure family asks whether a cluster wants to be two. A fixed two-means split gives clustering tendency, the fragmentation index, a continuous sub-cluster count and the silhouette, in Two-means as a measuring instrument.

The neighbourhood family places a cluster among the clusters around it: local gap, overlap, isolation, nearest distance, density, drift, cross-cluster similarity, member relevance, boundary clarity, topic alignment, phantom proximity and question-and-answer divergence, in Where a cluster sits. The Gaussian family treats the cluster as a field with a per-cluster length scale: potential depth, curvature, escape probability, superposition, interference and entanglement, in Physics as a lens on embeddings.

As one cluster
Split by two-means

From scores to decisions

Composite scores fold the families, with their weights held in a database table rather than in code, and operational metrics about a cluster's life in the database sit beside the geometry. Those scores decide what happens to a cluster, in From health scores to decisions.

Cluster A
splitCluster B
merged with its neighbourCluster C
left alone
Scores and thresholds are illustrative.

A CPU twin for every metric

Every metric is defined by a CPU reference, and the kernel is checked against it at a tolerance derived from single-precision reduction behaviour. The kernels, the references and the lineage documentation are checked against each other, in Every metric gets a CPU twin.

CPU reference value, with the tolerance band around it
the GPU kernel's value lands inside the band
Thresholds, seeds and iteration counts are part of the definition, so a value is comparable across the corpus and across runs. Scale is illustrative.

Metrics stay unclamped where their sign is informative, and every degenerate case has a written answer. A metric that would need a hidden size limit on the GPU stays on the CPU with an explicit limit until a bounded kernel exists.

A number means the same thing wherever it is read.