Corey James- 256 MBa segment rotates at this size, or when it goes idle
- 4 bytesthe in-place write that retires a work item, with no compaction
- 12.3Btokens received and routed, a 7B dense FP8 workload on two Blackwell GPUs
I designed completion accounting on the broker's log so that work retires only when acknowledged, which is how a six-million-item loss was closed. I fix data loss at the definition that caused it and build recovery on one source of truth.
The log is the queue
Inference Fabric needed producers to drop millions of tokenised tasks and walk away while GPUs on other machines worked through them for hours. I made the broker's backlog an append-only, memory-mapped log on NVMe and made that log the only durable copy, with no write-ahead log beside it. The log is the queue, which is what lets producers submit and exit.
The log is divided into segments, and each segment holds work for exactly one model, one prompt shape and one length lane. There is no per-item index; the broker tracks positions per segment. The hot path is one owner and a few blocking threads, chosen over an async runtime because it measured faster and is easier to reason about.
prompt shape
length lane
Dispatch and completion
Dispatch is two-phase. The commit checks every segment it will touch before it changes anything, so it cannot fail halfway, which gives failure atomicity without a transaction log. Each item carries its segment and byte offset to the GPU and back.
When a result comes back, the broker rewrites that entry's marker in place, then increments the segment's deleted count. That costs bookkeeping on the read path, and I took that cost over a fast path that once lost millions of items.
Results are held in a completion log with the same structure, and a result is marked delivered only after the response carrying it has been flushed to the receiving socket. A segment whose entries are all deleted is removed by a collector that never deletes a file someone is still reading.
Recovery from the markers alone
Recovery reads the deletion markers and nothing else. A segment with every entry retired is finished; any other segment is replayed from its start. An earlier design also consulted a separate dispatch file, and in a crash window the two could disagree, so I made the markers the single source of truth and recount the header from them on startup.
When a broker restarts while GPUs still hold work, each reconnecting GPU worker reports the work it is holding and the broker notes it before sending anything new, so surviving work is not run twice. When a worker disconnects or an in-flight entry goes stale, only the affected segments rewind to the earliest reclaimed entry. This is the model I filed as Family Two: durable work, tombstones and lossless recovery.
memory
1 output persisted · 2 the tombstone written · 3 the input retires
Computation can repeat after a failure; observable completion happens once.
The diagnosis that led here is in Millions of queued items that were not there.
Journalled shards in the vector store
Inference Fabric's Vector Shard Manager keeps vectors outside the database in append-only shards, each stored as payload, ids and a journal. The store does no deduplication; each producer gets back a stable reference to its payload, and an acknowledged append becomes durable at the next scheduled flush.
- mid-build shard: journal replayed and truncatedrecovered
- reopened shard: alignment rules checkedchecked
- connectionsonly after