Technology · Discipline

Large-scale GPU data pipelines

A generation pipeline that populated tens of millions of dataset items on two consumer GPUs, with persistent item identities, lease-based claiming and prompts baked at build time for cache reuse.

Rust on the FAFO Core substrateGeneration and hidden-state captureTwo NVIDIA Blackwell GPUs
Work items · each with a persistent identity
written, atomically
claimed under a lease
waiting
Blackwell GPU
pulls leased work
Blackwell GPU
pulls leased work
  • 876Kprobes submitted in one captured window
  • 131 Mtokens pending in that window
  • 156/sprobes per second, with zero retries
  • ~203Khidden-state extractions over 10.8 M tokens, 7B dense model in FP8

Generating tens of millions of traceable items

FAFO Core and Function Forge: my distillation research needed tens of millions of model-generated dataset items. Each one had to trace back to the knowledge it came from, and they had to be produced fast enough to keep two consumer GPUs saturated. I treated that as an industrial data pipeline, and I let the workload itself decide what the substrate needed.

The probe generator is a Rust function on the FAFO Core substrate. It reads a unified work view over three source types, assembles class-specific prompts, submits them to the GPU broker, and stores each result as a token-ID array with provenance. It runs in service mode with queue-depth triggers.

three sources
work items, each leased
broker
results, atomically

Each object arrived when the pipeline hit a wall: crashes brought leased identities, deadlocks the atomic writes, a producer waiting on results the separate completion service, a broker loss the recovery of its buffered backlog.

Every work item has a persistent identity and is claimed by lease. Two earlier designs put completion inside the producer, which made it wait on results; splitting completion into its own service let producers and GPUs run at full speed. See The workload that shaped the platform.

Prompts baked at build time

Prompts are tokenized at build time and baked into the binary, so every class shares a byte-identical prefix the inference engine can reuse from cache.

Requests in one class
Requests in another class
Pre-tokenized class prefix, byte-identical, served from cache
The part that varies per item

Baking them in made the build depend on an extra step. It also made prefix-cache reuse dependable, and I took that trade.

Hidden-state capture that can be rerun

Inference Fabric: the second stage captures a model's hidden states for every probe. I wanted that capture to be deterministic and safe to rerun.

A research result that cannot be reproduced is worth little.

A mining run checks its extraction spec against what the engine declares it can do. It freezes its probe inventory in the same transaction that creates the run, and content hashes over a canonical encoding make the run content-addressed, so resubmitting after a failure costs nothing. A TensorRT engine variant built without a key-value cache returns one last-token hidden state per probe.

One captured hidden state · as it travels
Envelope
typecontent hashrun
binary body, one last-token hidden state, opaque to the broker
Where it lands
  • Vector shardappended
  • Catalog row in PostgreSQLwritten
  • Failed probe, up to an attempt ceilingreplayed
The broker moves bytes it does not understand; meaning lives in the hash and the catalog.

An eight-state lifecycle exposes status, stalls and replay as MCP tools, so I can start, watch, cancel or replay a run from one conversation. The capture path carries a single layer per probe, because its inline payload size caps extraction there.