Technology · Discipline

Lease-based work claiming

Parallel workers claim work by lease, designed against the dead-tuple behaviour of PostgreSQL's concurrency model, so two workers never process the same item and nothing is lost on a crash.

PostgreSQL · RustFAFO Core conductorTens of millions of work items
Workers · each item under a time-bounded lease
Worker onelease renewed
Worker twolease held
Worker threecrashed, lease expired
Claimablerecovered, no item lost
  • 0.7–1 Mitems in flight at once over a two-hour window
  • 3ways the conductor slices a job; leased claiming is one
  • 2NVIDIA Blackwell GPUs fed by the claimed work

Claiming designed against dead tuples

FAFO Core and Function Forge: my probe-generation workload had parallel workers pulling rows of work from PostgreSQL, and two of them collided on the same item. I needed claiming that holds up under tens of millions of items and survives a worker dying mid-batch. I did not want to pay for it with database churn. The obvious pattern, updating a row to claim it and updating it again to release it, leaves dead tuples behind on every cycle, and at that scale the vacuum load becomes its own problem.

One work row under the obvious pattern · update to claim, update to release
One cycle Ten cycles Tens of millionsthe vacuum load becomes its own problem
The current row version
Dead versions left for vacuum

I added claiming when two workers collided on real work; nothing about it was speculative. I chose the pattern for what PostgreSQL does with an update under concurrency, and how a claim reads in application code came second. The claiming method itself is one of the parts of FAFO Core I keep private, along with the governed write path and the admission controller.

Claiming in the framework, not the function

In the generic conductor, the framework owns claiming, not the function.

How a job is sliced
By row count
Token-budget buckets
Lease-based claiming
Who writes what
framework: connections, slicing, dispatch, claims and leases, atomic job state, dead-lettering
function: the processing step

Every function on the substrate is crash-safe without implementing anything.

I design concurrency against the real costs of the database underneath, and I put the hard part in a framework so every function inherits it.

Leases, crashes and late signals

A worker holds a time-bounded lease on the items it claims. A worker on a long batch extends its lease. Claims left behind by a worker that crashed are recovered and become claimable again, so a crash neither loses an item nor lets two workers process it. Each worker signals ready exactly once per dispatched slice, and the conductor ignores stale signals, so a late message from a dead worker cannot confuse the count.

Leases over time · drawn, not to scale
Worker one
leasereneweddone
Worker two
leasecrashexpires
Worker three
reclaimed

The claiming went in alongside atomic multi-table writes, at the point where the pipeline needed both. The full account is in The workload that shaped the platform, and the database-side diagnosis habit behind it shows in 196 deadlocks to zero.