Corey James- 16 × 64pair tiles, one atomic per tile for the minimum and maximum
- 20histogram bins in shared memory, flushed once per block
- 5 Bpairwise distances recomputed rather than stored
- 1entropy score for a perfectly even spread
A cluster of embeddings can be tight and still be useless, because every member says the same thing. The diversity family measures the opposite of redundancy: how much of the cluster is distinct, how evenly its members are spread, and how broad the concept it covers is. Five of those metrics come from pairwise distances, and one of them, entropy, forced the design into three kernels where one would have been nicer.
The problem with entropy
Entropy of the pairwise distances needs a histogram, and a histogram needs its bin edges before the first distance is counted. The edges come from the minimum and maximum distance in the cluster, which are not known until every pair has been visited.
One pass cannot do both.
Pass one: statistics
The first kernel walks the cluster's pairs and accumulates nine quantities per cluster. The minimum and maximum are the awkward ones. CUDA has no atomic minimum or maximum for floating point, so the library carries both as compare-and-swap loops on the bit pattern.
- Pair countevery value
- Sum of distancescoverage
- Sum of squared distancescoverage
- Sum of cosinesaccumulated
- Sum of one minus cosineconcept breadth
- Redundant pairs, cosine above 0.9redundancy
- Dense pairs, cosine below 0.85information density
- Minimum distancehistogram edge · compare-and-swap
- Maximum distancehistogram edge · compare-and-swap
Pass two: the histogram
With the range known, the second kernel recomputes every pairwise distance and bins it into the histogram. A degenerate cluster whose maximum equals its minimum exits early with a documented result rather than dividing by zero.
Pass three: the metrics
The finalize kernel turns the accumulators into five values per cluster. Entropy is the Shannon entropy of the histogram normalised by the logarithm of twenty. Coverage is the coefficient of variation of the distances. Concept breadth is the mean of one minus cosine over all pairs.
Why the thresholds are fixed
The thresholds at 0.9 and 0.85 are part of the metric definitions, written into the CPU reference and the lineage file, and no run can change them. A diversity score that depended on a tunable threshold would not be comparable between runs, and comparability is the point: the pressure model that decides whether to split or merge a cluster reads these values across the whole corpus.
What this demonstrates
Letting the mathematics dictate the kernel structure. Two passes exist because entropy needs its range first, the float atomics exist because the hardware lacks them, and the thresholds are frozen because the values must be comparable.