Technology · Discipline

Distributed job orchestration

One function definition runs as a job or a service, batched by rows or by GPU token budget, anywhere from in-process to rented cloud GPUs, by configuration.

FAFO Core and Function ForgeRedis streams and LuaPostgreSQL
One function, three independent axes lifecyclejobservicebatchingrowstoken budgetsubmit and exitlocationin-processlocal containerLAN workstationcloud GPU
  • 13function crates built on the framework, beside eight service crates
  • 19control actions reaching running conductors, among them pause, scale, drain and inspect
  • 970Kprobes in flight at the peak, never below 670K, from one decoupled producer over a two-hour capture

One trait, three axes

I was going to build many systems and did not want to solve orchestration more than once, so I made FAFO Core and Function Forge one execution model that every function runs under. A function implements one Rust trait, and a generic conductor runs it along three independent axes over Redis and PostgreSQL.

Orthogonal axes over named modes: a new combination is a configuration change.

Services trigger continuously, on queue depth or on an interval. In submit-and-exit, a producer hands generation to the GPU broker and moves on while a completion service closes the loop; I split completion out after an embedded design made producers wait on results.

embedded completionproducersubmit and exitproducerGPU brokercompletion
Timing illustrative. Orange is the producer waiting on results.

The framework owns the hard parts

The framework owns connections, pagination, slicing, worker dispatch, dead-lettering and metrics, and it changes job state in Lua on the Redis server, so state is never half-updated between round-trips. Workers follow an "exactly one ready signal per dispatched slice" protocol.

slices dispatchedready signalslate: slice already settledduplicate
A late or duplicate signal cannot cause double dispatch.

Submit-and-return workers use a lock-free pending tracker and per-job ready queues, so polling cost follows the number of ready items. Work items are claimed by lease, a rule that came from two workers colliding under the real workload. Schemas, views and column types resolve at run time, and identifiers are validated and quoted before any dynamic SQL.

A control plane above the conductors

Commands arrive on a Redis stream and control actions on a separate one, and on startup the control plane recovers orphaned services and stale queues.

command streamcontrol streampausescaledraininspecttyped envelopes, consumer group, pending to processing atomically
A new function becomes controllable after a rebuild and a human restart, a step I kept as governance.

Under load, a decoupled producer keeps its probes in flight by refilling in waves across Blackwell GPU hosts; see The workload that shaped the platform.