The workload that shaped the platform

My distillation research needed tens of millions of model-generated dataset items, each traceable back to the knowledge it was drawn from, produced fast enough to keep two GPUs saturated.

I did not design a platform and then look for a workload. I ran the workload and built exactly what it forced.

The challenge

Dataset construction at this scale is an industrial data pipeline. Every item needs a persistent identity so a crash does not lose it or run it twice. Writes span several tables and must stay referentially consistent. Prompts have to be shaped so the inference engine can reuse cached work across millions of near-identical requests. And none of it may wait on anything else.

Constraints

Two consumer GPUs. Nothing waits: a slow writer must not stall generation, and a busy GPU must not stall producers. Foreign-key-safe writes across multiple tables under heavy concurrency. Crash safety, so work is neither lost nor duplicated across a restart of any component.

My approach

Let the real workload drive the substrate. Each time the pipeline hit a wall I added the one capability the wall demanded. Nothing speculative went in. What came out was a general execution substrate shaped by a job that ran.

The workload's run · each orange line a wall

wasted capacity → token-aware packingtwo workers, one item → leased claimsdeadlock → atomic multi-table writescompletion blocking → decoupled producer

What I built

A probe-generation library with explicit input and output contracts and reviewed phases, and the function that runs it on the substrate.

Token-aware packing, so each batch submitted to the GPU broker carries similar shapes and minimal padding.

Lease-based claiming of work items, so parallel workers do not process the same item.

Multi-table atomic writes that keep the result tables consistent under load.

A decoupled submit-and-exit producer. Generation requests go to the GPU broker and the producer moves on; a separate completion service closes the loop. Two earlier attempts at embedding completion in the producer are kept as the iteration history.

A build step that moves a runtime dependency into the binary.

Recovery of buffered items after a broker loss, so in-flight work survived an infrastructure failure.

Difficult decisions

Moving a runtime dependency into the build. It made the binary depend on a build step, and it was what made cache reuse dependable.

Decoupling completion. The embedded design was simpler to explain. It also made the producer wait on results.

The producer's time · each block is one submission

Completion embedded

waitwaitwaitwait

Submit and exit

Completion service

closes the loop on its own clock

Results

Items in flight over a two-hour window

0about 0.7 to 1 million

After an infrastructure failure, a large backlog of buffered items was recovered instead of regenerated.

Technologies used

Rust · PostgreSQL · Redis · TensorRT-LLM · Grafana · Large-scale dataset construction with provenance · NVIDIA Blackwell GPUs

What this demonstrates

I build platforms from workloads. The substrate under my later systems has the shape it has because a real job of tens of millions of items ran on two consumer GPUs and told me what was missing.