Technology · Hardware

NVMe (cold-tier storage)

The broker's cold tier is NVMe: billions of queued tokens survive a restart in shape-pure order.

Local NVMe, memory-mapped segmentsInference Fabric brokerFeeds NVIDIA Blackwell GPUs
RAM · only the hot working set
99.9%key-value cache hit rate from shape-pure reads
  • 0shape switches while draining one backlog
  • 0.42%padding waste across that backlog
  • 2NVIDIA Blackwell GPUs fed from one cold tier

A cold tier laid out for how the GPUs read

Inference Fabric put a broker between producers that submit millions of tokenised tasks and walk away, and GPU consumers that pull work as fast as they can process it. The two sides run at different speeds for hours. Producers burst far ahead of the GPUs, and the GPUs must never wait on a producer.

A backlog of billions of tokens does not fit in memory, and losing it on a restart would mean regenerating hours of queued work. I gave the broker a cold tier on local NVMe: an append-only store of memory-mapped segments, with only the hot working set in RAM, so disk sets the backlog's size. Append-only writes are sequential, the access pattern NVMe handles best, and the backlog outlives the process that wrote it.

I organised it around how the GPUs read. The broker groups incoming work by token shape before it reaches disk, so each segment holds work of one shape, and when a consumer asks for a batch the broker reads from segments that already fit that consumer, in the order it wants. Nothing ever scans the backlog to find a fit. That layout is what let one broker drain a single backlog across both GPUs. See A message broker that understands tokens.

Arriving from producers · mixed shapes
On NVMe · one segment per shape

I treat local NVMe as a deliberate tier in a throughput system, laid out for the way the consumer reads.

Knowing what the storage holds

Durable storage covers half of keeping work. The other half is knowing what the storage holds. When the broker's registry and its storage disagreed by millions of items, I redefined an item as done only when its result has been delivered and acknowledged, and the accounting panels then balanced. Dashboards show what the storage holds next to what the broker believes, at any moment. See Millions of queued items that were not there.

When an item counts as done
  • Written to a segmentnot done
  • Handed to a consumernot done
  • Result delivered and acknowledgeddone
Accounting after the change · tokens
Received12.3BRouted12.3B

I also took length filtering out of the broker. The runtime returns an oversized item to the broker, and nothing strands in storage. The drives, file system and memory mapping are standard parts; the cold-tier design, its organisation and the accounting around it are mine.