Corey JamesA crash between making a call and logging it is the oldest problem in distributed systems, and the default answer, retry, is how you charge a customer twice, post the same message three times, or run the same inference on two GPUs. My systems do not retry blindly. They mark an effect as uncertain, reconcile against the real state, and resubmit only what is proven missing, under the same identity.
The shape
Before an effect, a durable entry for the intent. After the effect, a durable entry for the outcome. If the process dies between the two, the entry is in an explicit uncertain state, and recovery reads reality, the provider, the GPU worker, the receiver, before deciding. Nothing is silently retried and nothing is silently dropped.
Where it appears
In the Inference Fabric, completion is a durable tombstone written in a fixed order, so a crash between steps is repaired by a scan: Millions of queued items that were not there.
In FAFOHive, a restart does not repeat a provider call: Agents with real tools and no keys.
In AgentOS, resuming a work order after a stand-down is idempotent, with a stale generation explicitly superseded.
In Helix, the control core fences generations with execution leases so a pending intent is reconciled rather than re-run.
What it buys and what it costs
It buys no duplicated payments, posts, renders or GPU work, and no silent loss. It buys a recovery path that can be reasoned about, because every uncertain effect has an entry.
It costs throughput and sometimes a human. Some actions stay uncertain until an operator decides, and that is the point.
An external effect that cannot be confirmed is not something a retry loop should decide on its own.
The judgment in it
The failure that taught me this was a broker that counted a read as a completion. The fix was a definition, and the definition is now everywhere a side effect crosses a boundary.