SQLite

One database per social module

FAFOHive's in-house social modules each need durable, transactional state owned by one process, and I gave each its own SQLite database. Where one process owns the state, most concurrency turns into ordinary transactions.

The X module keeps a work queue in SQLite and claims jobs inside BEGIN IMMEDIATE transactions, which take the write lock up front, so two workers never claim the same job after a scheduler restart. The Facebook module keeps an inventory of every object it creates, which gives cleanup and reconciliation an exact list. The Reddit module stores a receipt for every change and an archive of every response it reads, so a repeated request with the same idempotency key returns the first receipt.

Two workers after a scheduler restart

Worker A

Worker B

lock released

A holds the write lock and claims a job; B waits, then claims a different one

One request, then a repeat with the same key

First

Repeat

change made, receipt storedthe first receipt returned, no second change

SQLite is a library inside these services, run with write-ahead logging and explicit transaction modes; the engineering is the state design.

Render jobs written down before they are sent

The media studio behind FAFOHive's media tools uses the same pattern for GPU rendering jobs. Each submission's identity is written to SQLite, under a uniqueness constraint on actor and request, before the job goes to the renderer, so a crash cannot leave an untracked job.

A job whose outcome is unknown, because a process stopped between sending and hearing back, sits in an explicit outcome-unknown state and is settled by reading the renderer at startup. The gateway these modules join is described in Agents with real tools and no keys.

Render job

identity writtenprocess stopsread from the renderer at startup

the job is never sent again

I pick the smallest storage that gives a component its guarantee, and I design around the moment between sent and confirmed.