The database decides when an index goes stale

An approximate index is a summary of the vectors it was built over. When those vectors change, the index is wrong, and a service that keeps serving it will return confident, stale neighbours with no error. The usual defence is a rule in the application: remember to rebuild. I put the rule in the database instead, where it cannot be forgotten, and backed it with checks at load time and at startup.

Two triggers in the database

Vectors live in append-only shards, each with a journal whose entry count advances on every append. A trigger on the catalog marks every active index that covers a shard as stale the moment that shard's journal count advances, with a reason attached. A second trigger fires when a newer generation of payload shards is activated and marks the indexes built over older generations as stale.

Trigger one · a shard's journal advances

Shard aShard bShard c

index over a and b · stale

index · active

Trigger two · a newer generation is activated

Older generationNewer generation

index · stale

The orange page is the append; the trigger marks every index whose bar spans that shard, with its reason.

The write that changes the data is the write that invalidates the index.

Three checks at load time

Before an index file is trusted, the loader checks that the kernel version it was built with matches the kernels that will search it and that its configuration hash matches the configuration being requested. Only then is the artifact hash checked against the file's bytes.

Before an index file is trusted

Kernel version matches the search kernels

Configuration hash matches the request

Artifact hash matches the bytes

Trusted and searchable

Each orange gate refuses with its reason; nothing stopped at a gate is ever searched.

Startup before the port opens

A build that was in progress when the daemon died has a catalog row saying it is building and a partial file on disk. On startup, before the listener binds, every such row is failed with a reason naming the restart. The step never has to be remembered.

Daemon startup · in order

Restart

Sweep: building rows failed, partial files removed, each auto-failure counted

Listener binds, first connection

Catalog row, before

Build in progressbuilding

and a partial file on disk

Catalog row, after

Build in progressfailed · restart

partial file

What a reader gets

Search looks up the active index for a shard set, loads and checks it, and falls back to the general index when no lattice index is active. A stale index is never a candidate. A build that fails emits a failure event on the same channel as a success, so the absence of an index is as visible as its presence.

Shard set with an active lattice indexlattice index

Shard set whose index is stalegeneral index

A failed buildfailure event

What this demonstrates

Correctness rules placed at the layer that cannot forget them: the database for data changes, the loader for compatibility, and startup for crash residue, so that the service either serves a current index or says why it cannot.

The placement of the rules was my decision; AI coding agents, a mix of frontier and open-source models, implemented the triggers, checks and recovery under my governance system.