Technology · Discipline

Distributed-systems reliability

Nothing waits on anything else, every component restarts independently, and the accounting balances.

Inference Fabric buffer serviceAgent SwarmFailure-domain model
Before · a full channel dropped the item
Now · a full channel only delays it
delivered and acknowledgedlost on a full channel
Tokens received and routed in one capture under load.
12.3B = 12.3Ba difference of zero
  • 32at-most-once windows written into the gap register
  • 3,099batches whose fill once blocked the broker for 17 seconds
  • 6leakage rules binding the delivery, coordination and durability planes

Done means delivered and acknowledged

The Inference Fabric lost work silently under load. Its buffer service is a custom distributed system: producers, a durable broker and pull-based GPU consumers on separate machines, each restartable on its own, with no distributed transactions and no shared state store on the hot path.

What the broker believed against what existed
Registry: pending6.67M Storage: held118K

I traced it to a definition. An item counted as consumed when it was read and was dropped if the consumer's channel happened to be full. I redefined done as delivered and acknowledged and rebuilt the accounting around it, accepting bookkeeping cost on the read path, so a full channel delays work and loses none.

“Consumed on read” was the bug, and “acknowledged on completion” was the cure.

A broker restart could also re-run work the GPUs still held, and I closed that path while staying compatible with consumers already in the field.

Liveness that heavy work cannot starve

The second failure was zombie consumers. Filling a large backlog blocked the broker's single-owner actor long enough that heartbeats starved and working GPUs were declared dead. The fix was a time budget on channel filling, one disconnect rule for every socket error, and deletion of a legacy path with no heartbeat timeout.

One actor, one backlog Before · one long fill
After · filling in budgeted slices
channel filling
heartbeat handled

Accounting that balances

Downstream, the completion service stops polling when the database write queue is unhealthy and backs off exponentially, so results never leave the broker with nowhere safe to land. Accounting panels show tokens received, routed, completed and acknowledged side by side. I chose not to pay for exactly-once delivery, and wrote down the windows that remain. See Millions of queued items that were not there.

work itemsat-most-once windows, each written in the gap registerexactly-once delivery: not paid for
Window widths illustrative.

Failure domains named before transport

In Agent Swarm, agents on several hosts must keep talking when a coordinator goes down, so I wrote the failure-domain model before any transport code. The unit is a swarm, a group of containers with its own Redis, and a host is disposable compute; every component knows what it may depend on.

deliverycoordinationmaster downdurability
Timing illustrative. The hot path never needs the central database.

The cross-host relay and hub router carry signed envelopes, deduplication, resume and forged-origin rejection.