Corey James- 16scripts in the server-side library
- ≈4,000lines of Lua, loaded by hash
- 3Redis instances, one per role
Job state that cannot be half-updated
The FAFO Core and Function Forge control plane keeps job state in Redis, and many workers change that state at once. A Redis pipeline batches commands without making them atomic, and a crash or a competing worker between two commands leaves a job half-updated.
I moved every multi-step state change onto the server as a Lua script. Job promotion is one script: it moves a job from pending to processing and writes the job's worker count, batch size and batch mode in the same step, so no worker ever sees a half-promoted job.
Scripts that return what they changed
When a worker finishes or fails a slice, one script updates the job's slice and item counters together and returns the new job state in the same call. The caller acts on the state the script produced and never on a separate read that may already be stale, which removes a read-after-write race from every caller.
The scripts are loaded by hash, kept in one directory with their own management tooling and hot-reloadable, alongside scripts embedded in the Rust core, so the control plane's logic is versioned like code.
One pattern, two places
The governed write path and the admission controller in PostgreSQL Fabric also keep their bookkeeping in Lua on the server. The write path gives many producers one governed way into PostgreSQL, and the admission controller keeps unbounded agent demand within the database's capacity.
One pattern travelled. A fenced single-flight script first written for admission was reused to mint sub-agent identities atomically, so a child's identity cannot be issued against a parent that has just been revoked.
The database side of this control plane is in 196 deadlocks to zero, and the workload that pushed it hardest in The workload that shaped the platform.
Redis split by role
The Inference Fabric splits Redis into separate instances by role, which keeps one workload's backlog out of another's memory.
A loader service installs the script set at startup, and the fabric's promotion and batch-claim steps call those scripts. Their semantics belong to FAFO Core, and the fabric uses them as they are.
I use the smallest script that does the job. Lua here is a tool for atomicity; the substance is the state model it protects.
I find the places where distributed state can tear and close them at the server.