Corey James- 3kinds of interruption, each once a path to loss or duplicate work
- 1 rulerecovery believes only what physically happened
- 100 msbetween tombstone flushes to disk
A broker that holds tens of millions of queued items on disk has to survive three kinds of interruption: its own restart, a GPU consumer disconnecting mid-batch, and an item that was admitted for dispatch and then never heard of again. Each of those had, at one point, a path to permanent loss or duplicate work. The design that closed them rests on one rule: recovery believes only evidence of events that physically happened, and nothing that was merely intended. The durability model behind it is one of my filed inventions, Family Two, and this is how it behaves in the broker.
Tombstones are the truth
When a GPU completes an item, the broker writes a tombstone over that item's entry in its segment, then increments the segment header's deleted count, in that order. Tombstones are buffered and flushed about every hundred milliseconds, and a segment is removed when its deleted count reaches its entry count. On restart, recovery collects the segment paths and scans the segments in parallel, counting unfinished tokens per segment, and it deliberately ignores the committed offsets it had persisted.
The offset had advanced and the completion mark had not landed. On restart the item in the gap (orange) was skipped and lost.
A parallel scan counts what is still unfinished. The same item is read again, because nothing says it finished.
An offset says how far dispatch had reached; a tombstone says what was finished.
Rewind before you forget
An item that was admitted for dispatch but never written to a socket, or written but never finished, has to be reclaimed. A watchdog that simply deleted a stale pending entry would lose the item for good. The same rule governs a consumer disconnect: every item the consumer had in either state is collected and its dispatchable tokens are restored.
Light blue: the rewound span, back to the earliest reclaimed entry and no further. The rewind is bounded by what was actually outstanding.
A GPU that says what it still holds
The consumer SDK keeps the GPU working while the broker is away. A heartbeat thread watches the broker on a separate stateless port; while it is down the consumer carries on through its local queue, and reconnection uses exponential backoff. When it reconnects, it re-registers with the list of sequence ids it still holds, and the broker marks those as held before it dispatches anything new. Without that step a restarted broker re-sent work the GPU was already doing, and the same items ran twice. Sessions are keyed by a fresh identifier rather than a friendly name, after reconnects under the old scheme produced index mismatches, and timeouts stay disabled until the broker has finished loading, so a restart cannot declare healthy GPUs dead.
Hints, never gates
One starvation bug taught a rule that now runs through the dispatch path. An advisory structure that can veto dispatch is a bug waiting for the case it does not model.
The files that make it possible
Dispatch state is kept in append-only files with a header epoch, set at broker start so entries from an earlier life are ignored, and a checksum on every entry. Writes are synced in batches of thirty-two entries or a hundred milliseconds, and compaction rewrites a file without its resolved or duplicate entries. Those files serve disconnect handling while the broker runs; restart recovery uses tombstones alone.
What this demonstrates
Durability designed from the failure cases outward, with a single source of truth, rewinds bounded by evidence, and a consumer that participates in its own recovery. The accounting result is in Millions of queued items that were not there.
The recovery rules were my design; AI coding agents, a mix of frontier and open-source models, implemented them under my governance system.