Technology · Discipline

Catalog versioning and drift detection

Every host in my agent fleet runs the same approved tool catalog, verified by content hash, with drift detection that falls back from notifications to polling under measured thresholds and never flaps.

SHA-256 content hashesPostgreSQL LISTEN/NOTIFYRedis per host
Hosts against the canonical version
drift repaired in process, with no restart
30 sbetween drift checks on each host

I treat agent configuration with the same consistency engineering as replicated data.

One catalog version on every host

Agent Swarm changes personas and tools while agents run on several hosts, and I saw two ways that goes wrong: hosts silently diverge, and an old agent binds to a retired tool. I started with a catalog version that means the same thing everywhere.

the catalog's content
SHA
256
PostgreSQL rows
Redis
Redis
Redis
a persona pins a tool
older versionsnewer versions
tools
active
deprecated
The shaded range is a semver subset; the line is the version the host resolved. Illustrative cells.

Hot-reload actions on the persona launcher apply a new catalog without a restart, and no tool enters the catalog without its golden and attack fixtures. The approach is told in Tools are the platform.

Drift repair, and when to stop trusting notifications

A drift detector on each host compares its catalog with the canonical version and repairs a sustained mismatch by refetching and swapping the catalog atomically in process; only a sustained mismatch triggers repair, so a catalog mid-rollout is left alone. Changes normally arrive at once through PostgreSQL LISTEN/NOTIFY, and explicit backlog and lag thresholds decide when to stop trusting them. Polling everywhere would have been simpler and slower.

The first mark: backlog or lag crosses the switch threshold. The second: both are back under the lower thresholds and the hold begins.

notifications
short-interval polling
hold
Separate switch and return thresholds keep a loaded host from flapping between modes.

Frozen inputs for measurement

Capabilities model trains and evaluates a tool-routing model against a fixed catalog, where drift corrupts measurements. The synthetic distractor catalog was regenerated after a baseline had already been taken against it. I treated that as a defect.

at the baseline
regenerated
frozen, re-run
  • distractor catalog frozen under a checksumfrozen
  • affected measurements re-run against the frozen versionre-run
  • client catalog fingerprint differs from the engine's: the client refuses to startrefused
Orange cells changed under a baseline that had already been taken. Illustrative cells.

Across the serving socket, the engine reports a fingerprint of the catalog its decoder was built from. The evaluation discipline is told in The baseline that tied us.

Evaluation inputs are frozen by fingerprint, which ties every measurement to the exact catalog it ran against.