Cache synchronisation and leader election

I treat a cache that agents trust as replicated state with explicit rules for leadership, progress and deletion.

FAFO Core and Function Forge has a sibling platform I designed for putting any third-party or internal API behind a governed MCP server. Each API is described by a declarative connectivity profile, and agents read from a Redis cache of that API's entities, which spares the API a call on every agent request. That makes the cache the thing agents trust, so it has to follow the source. How it was built is told in Closed by evidence.

One leader per namespace

Only one process may drive the sync for a namespace at a time. I chose a timer-driven singleton sync worker per namespace, elected through a Redis lease that expires on its own. A leader that dies stops renewing, and its successor takes over at the next expiry.

W1

W2

Progress only on a full cycle

Each delta cycle fetches the entities changed since a watermark, a source timestamp. Reads are charged against the platform's read budget per page, metered by the sync worker and invisible to the connectors, so connectors stay budget-agnostic and the sync loop owns the cost in one place. A profile can use whichever change signal its API offers.

Delta cycles against the source's timestamps

every page readwatermark advances

a page failed, or the read budget ran outwatermark stays

the same work, repeatedwatermark advances

Deletion only from a complete view

A periodic full pass removes entities that no longer exist at the source.

Full pass completed

• entity gone from the sourceevicted

Pass cut short by budget

• entity not seen in a partial listkept

Eviction requires a complete view of the source; with an incomplete view the cache keeps what it has.