Physics as a lens on embeddings

Distances between embeddings are the raw material of every cluster metric, and most metrics use them directly. I wanted a family that treated a cluster as a field: each member a point charge, the centroid a probe, and the shape of the resulting potential as the description of the cluster. The vocabulary is borrowed from physics, the formulas are plain Gaussian kernels, and the metrics turned out to say things the direct statistics do not.

The length scale first

A Gaussian kernel needs a width, and a fixed width would make the metrics depend on the embedding model's scale. So each cluster gets its own characteristic scale: a constant times the median of its pairwise distances. The median of millions of distances is found without storing them. A tiled kernel bins every pairwise distance into a histogram in shared memory, and one thread walks the cumulative counts to the midpoint. Every metric in this family reads the result.

Every pairwise distance, binned

Grey: bins below the midpoint. The thread walking the cumulative counts stops at the coloured bin.

What the kernel holds

The bins live in shared memory, about 10 KB

Distances are counted, never kept

The scale is clamped to a documented range

Five metrics in one pass

With the centroid and its squared norm in shared memory, a streaming kernel computes five values in a single sweep over the members. Each asks a different question of the same field.

the cluster and its centroidbarrier to the nearest rival

Potential depth: the Gaussian kernel of each distance to the centre, summedmass
Curvature: the Laplacian of the potential at the centroidpeak or plateau
Escape probability: exp of minus barrier over scalerival
Superposition: one minus the coefficient of variation of distancesunclamped
Interference: mean cosine of each member with the centroidunclamped

Curvature separates two clusters of the same depth. A negative superposition or interference is information about the cluster, and a clamp would hide it.

Entanglement over pairs

Entanglement is the mean off-diagonal Gaussian kernel over all pairs of members, the one metric in the family that is quadratic. It runs as a tiled kernel over the upper triangle with the same tile machinery as the diversity family and never forms the Gram matrix.

The three that stayed on the CPU

Three spectral metrics need the eigenvalues of the Gaussian Gram matrix: the coherence index, the largest eigenvalue over the trace; the spectral width, the eigenvalues' coefficient of variation; and the peak count above a threshold. They exist as CPU reference code with a hard limit, because the Gram matrix grows with the square of the member count.

GPU · no size limit

scale, depth, curvature, escape, superposition, interference, entanglement

CPU reference only · up to 2,000 memberscoherence index, spectral width, peak count

Streaming kernels for the linear metrics; a tiled upper triangle for entanglement, with no Gram matrix formed.

I kept them out of the GPU count rather than put a kernel with a silent size limit on the card.

Why borrow the vocabulary

The names are a lens. Depth, curvature and escape probability are three different questions about the same cluster that a mean and a variance cannot separate: how much is at the centre, how sharply it falls away, and how close the nearest competitor is. Naming them after physical quantities keeps the questions distinct for the people reading the dashboards.

What this demonstrates

Designing metrics with an explicit model behind them and the engineering to make them cheap: a scale derived per cluster without materialising its distances, five field metrics in one pass, a quadratic one bounded by tiling, and three left off the GPU until they can run without a hidden cap.