Corey James- 1,024dimensions in every unit's embedding
- 8signals weighted into each cluster's pressure
- ~11,000lines of Rust in the sole acceptance authority for new clusters
My distillation corpus is a lattice: question-and-answer units, each with a 1,024-dimension embedding, grouped into concept clusters with a centroid, a tier and a pressure score. The lattice has to grow where it is thin, stay stable where it is dense, admit new clusters only on evidence, and never let its seed units move. I built those behaviours as functions that run inside PostgreSQL, because the work is a set of index-backed nearest-neighbour queries and one atomic update, and moving millions of vectors to an application server would cost more than the work itself.
Pressure
Every cluster carries a saturation pressure between zero and one, higher meaning more need to grow. It is a weighted sum of signals about the cluster's size, spread, history and neighbourhood. Missing signals default to the midpoint, and updates are queued and applied in batches.
Redistribution
One database function runs a whole schema as a single transaction. It selects targets: active clusters with a centroid, pressure above a threshold, an eligible tier, and no prior fill event. A cluster is filled at most once, which stops runs from oscillating units back and forth. For each target it runs a lateral nearest-neighbour query over units in other clusters, ordered by cosine distance to the centroid through an inverted-file index, limited to a candidate count and filtered by a similarity threshold. Golden seed units are excluded and never move. The lateral join is chosen so each centroid's search uses the index instead of a cross product.
Candidates are then ranked within their source cluster and blocked by four protection rules, each counted separately. The move is one update that re-parents the allowed units, and history rows log the fill on each target and the drain on each source with counts and a run id. The function returns clusters processed, units moved, candidates found and blocked, and the four blocking counts.
Each rule's blocks are counted separately. A dry run computes the same sets and moves nothing. Positions are illustrative.
A reverse mode works from the other side: units in high-pressure sources each find their single best centroid among eligible targets, under the same thresholds and protections. Three presets, conservative, balanced and aggressive, set the similarity and pressure thresholds, the candidate limit and the four protections.
Admission of new clusters
A candidate cluster, synthesised from a group of generated question-and-answer pairs, is scored deterministically, and every failed test comes back as a reason. This replaced a Python gatekeeper and is the pipeline's sole acceptance authority for new clusters.
Health
Cluster health is measured in three tiers of metrics. These feed the pressure model's inputs, and the GPU re-implementation of the same statistics is the engine in Thirty-eight metrics in a dozen passes.
What this demonstrates
Data-locality judgment: when the algorithm is nearest-neighbour queries and one atomic update, the database is the right engine, and making the protection rules first-class and their effects countable is what makes the system tunable.
Fill once, protect seeds, keep history, and the lattice converges instead of churning.