Corey James- 4 Mrows of embeddings the GPU files are built from
- 2paths, delta and full, that must write the same bytes
GPU workers read vectors from contiguous files on disk, built from the embedding columns of a relational database: one file of single-precision rows, one of identifiers, and for member vectors a table of offsets that maps each cluster to its range of rows. Rebuilding all of them because a few embeddings changed is a waste, and doing it on a schedule is worse, because age says nothing about whether an embedding is still correct.
Capturing the change where it happens
A trigger on each member table captures inserts and deletes, and captures an update only when the embedding column is different from its old value. Changes land in a pending table and are grouped into batches, so nothing is lost between builds. An evaluation timer runs on its own cadence.
A listener wakes the build daemon on a new batch, with polling as the fallback.
Full or delta, by shape alone
The decision is a function in the database. There is no age term. A setting that would have forced a full rebuild after a fixed age exists in the configuration and is read by nothing, which is the honest state of a rule I decided against.
Either meter past its line, or no active generation, means full; nothing pending means none. Meter fills are illustrative.
A merge that reproduces the full build
A delta build must produce exactly the file a full build would have produced, or the two paths diverge and every consumer has to care which one ran.
The delta builder loads the active file's cluster offsets, fetches the identifiers of the clusters that changed, prefetches fresh rows for those clusters only, and then walks every cluster in sorted order. The output keeps the full build's ordering, cluster then row, so a delta build and a full build of the same data are the same bytes. The builder reports how much it copied and how much it regenerated.
What the capture costs
The capture trigger is declared on every update of the member table, so its function-call overhead is paid even when no embedding changed; declaring it on the embedding column alone would avoid that.
What this demonstrates
Incremental rebuilds designed around a correctness invariant, byte identity with the full build, with the decision rule kept simple and shape-based.
The byte-identity rule and the no-age decision were mine; AI coding agents, a mix of frontier and open-source models, implemented the capture and the merge under my governance system.