Technology · Discipline

Idempotency, exactly-once effects and crash recovery

A restart never repeats a provider call, a broker restart never re-runs work the GPUs hold, and an agent's claimed completion is a claim.

AgentOS · Helix · Agent SwarmFAFOHive · Inference FabricDurable state in PostgreSQL
The crash window
intent and budget committed
external call in flight
uncertain: an explicit state
Resolve by readinga read-only check, or a person
Retry by sending againnever

The rule I apply in every system below is simple to state.

An outcome I cannot confirm gets reconciled by reading, never retried by sending again, and work is done when its result has been acknowledged.

I design for the crash window explicitly and decide, per system, whether a duplicate or a gap is the worse failure.

No repeated payment, no second post

FAFOHive lets agents use a payments provider, an email/CRM provider and social-publishing providers through a governed gateway. A repeated payment or a second marketing email is a real-world harm, so I made the gateway account for every call durably, and a restart does not repeat a provider call. The gateway's design is in Agents with real tools and no keys.

In my own social modules a duplicate public post is worse than a missing one. One service owns the send log, and intent is persisted before a side effect: a durable sending state and a budget reservation are committed before any external call. After a crash, anything that was mid-send becomes uncertain, a first-class state that stays until a read-only check or a person resolves it. The Reddit module binds each request to an idempotency key that binds its inputs.

First request
sent; receipt saved
Same inputs again
the saved receipt, no call
Changed inputs
refused
Every change is then read back from the published page.

Renders that are never produced twice

Helix keeps durable state for conversations and the work agents do inside them. Its store uses idempotent lifecycle operations and an append-only event log, and a worker that has been replaced cannot overwrite the state of its successor. A pending-intent reconciler finishes or closes out work a crash interrupted. Documents agents create in a conversation are immutable versions with optimistic concurrency, so two writers cannot silently overwrite each other.

In the agents' media studio, renders are long and expensive and a dropped response must never cause a second one.

123456
1 identity persisted2 submitted3 rendering4 outputs registered5 response dropped6 reconciled, delivered
a second render: never resubmitted. Until reconciled the agent sees an explicit outcome-unknown state.
Keyed by caller and request id.

One request, one agent

Agent Swarm must never start two agents for one request. The writer that moves agent activity into PostgreSQL deduplicates at two layers and checks the body of any replay.

Launch, with its claim
one job
Replay, same context
the same job back
Replay, other context
fails closed
Settlement, twice
→settles once, deterministic id

Built to be interrupted

AgentOS is built to be interrupted. A crash, a lost session or a context compaction leaves the work order intact. See Why long agent runs fail.

the work order, held in the databasecrashlost sessioncompaction
State clear
resume, idempotent; the team re-grounds
State ambiguous
refused; an operator supersedes, logged

Counted when acknowledged, not when read

Inference Fabric had a broker that counted work as consumed when it was read. That is Millions of queued items that were not there.

Before
queued
read · counted done
delivered
acknowledged
After
queued
read
delivered
acknowledged · counted done
Now a full channel only delays work, and a restart leaves alone what the GPUs hold.