Corey James- 196 → 0deadlocks, before and after one schema change on the shared path
- 2 sthe slowest batches before that change
One governed write path
FAFO Core and Function Forge runs functions that fan out to many GPU workers on several machines, each producing results in bursts. If every worker wrote to PostgreSQL itself, the database would see as many writers as there are workers, each holding connections and locks while it waits on a GPU. I made a house rule.
Functions read from PostgreSQL directly, and every write leaves the function as a message that one governed writer applies.
Workers hand off results and move on, and the writer, not the workers, sets the database's write load. Schema validation lives on that single path and nowhere else, so a malformed row is rejected in one place for every function. A function that writes to several related tables hands off the set together, so related rows land as a unit.
Results for producers that walked away
Inference Fabric feeds the same path. Producers there submit inference work and exit, sometimes hours before results exist. I built a completion service that receives results from the GPUs and dispatches them per function into the governed write path, so a producer that walked away still gets its results stored.
Judged by rows committed
When a bulk update workload slowed that shared write path to a crawl, with frequent deadlocks, I measured from the database's own statistics, because the queue in front of it counts intake and the database counts committed rows. Comparing runs before and after a single schema change ended the deadlocks. See 196 deadlocks to zero.