Millions of queued items that were not there

The broker's registry and its storage disagreed by millions of items. Somewhere between a producer handing work in and a GPU finishing it, that work had ceased to exist without a trace. I traced it to a definition: the broker counted an item as consumed the moment it was read, and dropped it if the consumer's channel happened to be full. I redefined "done" as acknowledged on completion, and rebuilt the accounting around that until received and routed agreed to the token.

The challenge

Two problems with one root. Work disappeared under load and nothing reported it: the registry believed the items were pending, the storage no longer had them, and the GPUs had never seen them. And when the broker restarted it forgot what it had already dispatched while the GPU workers still held that work, so a restart risked running the same inference twice. The broker's idea of where work was did not match where work actually was.

Items pending · before the fix

Registry said6.67 MStorage held118 K

Where the gap came from

counted as consumed the moment it was read

dropped when the consumer's channel was full

the GPUs never saw it

Constraints

A busy GPU fleet, so the fix had to go in under load. No distributed transactions and no shared state store on the hot path; the broker's speed came from having neither. Backward compatibility, because older consumers had to keep working against the new broker.

My approach

Fix the semantics. An item is done when the result has been delivered and the receiver has acknowledged it. Everything else is in flight, and in-flight work must survive both a full channel and a restart.

Before

Read · done

Queued

Delivered

Acked

After

Read

Queued

Delivered

Acked · done

What I built

Accounting rebuilt around acknowledged completion, so a full channel delays work instead of losing it and a restart no longer re-runs work the GPUs already hold.

Accounting panels that show tokens received, routed, finished and acknowledged side by side, so a discrepancy is visible within seconds instead of after millions of items.

Difficult decisions

Paying for bookkeeping on the read path. Completion-only retirement costs more at read time than retire-on-read.

The alternative was a system that was fast and occasionally lost millions of items.

Trusting evidence of what physically happened in recovery, and never the broker's own last known state.

Documenting the residual windows. There remain small windows where at-most-once delivery is possible. I did not pay for exactly-once semantics; I wrote down where the windows are.

Results

Before

Received

Routed

vanished, unreported

After

Received

Routed

Technologies used

Rust · Append-only memory-mapped logs · TCP protocol design · Prometheus accounting panels

What this demonstrates

I root-cause data loss in a custom distributed system from a quantified symptom, fix the definition that caused it rather than the place it showed up, and leave behind a balanced count and a written map of the risk that remains.

The recovery design is one of my filed inventions, described on the research page.