Engineering story · Capabilities model

The recovery that drained the card

Four failed engine loads took a shared GPU from 1,801 MiB free to 2, because partial allocations were never released; the fix made creation crash-safe and made the admission arithmetic match the allocation it predicts

CUDA C++ and RustCapabilities modelOne shared GPU
Free memory on the shared card, after each failed load
2 MiBleft beside a larger serving model
  • 4failed engine loads in a row, each one passing admission
  • ~503 MiBthe CUDA context the admission arithmetic had left out
  • 2,740MiB, the engine's whole allowance beside a 30B serving model

The tool-discovery engine lives on a GPU beside a 30-billion-parameter serving model, in a fixed allowance and no more. An admission check compares the engine's requirement with the card's free memory before anything is allocated, and refuses with the shortfall if it does not fit. That check once passed every time while the engine failed to load every time, and the card drained.

What went wrong

Engine creation allocated its buffers in sequence, and when a later allocation failed, the earlier ones were not freed before the error returned. Each retry therefore started with less free memory than the last, passed admission because the arithmetic looked only at the engine's own total, and failed further along. A larger serving model was sharing that card.

Before · a failed load keeps what it took

Four attempts, each stranding its partial buffers.

After · four failed loads, nothing held

Every allocation before the failure is released before the error returns.

Shaded squares are GPU memory still held after a load failed.

Three fixes

A real loadAdmission's prediction, mirroring the allocatorThe old prediction, with the context left outlengths illustrative
1 · CreationCrash-safenothing allocated survives a failed load
2 · AdmissionA mirror of the allocatorsame parameters, and a test that compares the prediction with a real load
3 · ContextCounted on its ownleaving it out made admission optimistic by the card's whole margin
The prediction and the allocation cannot drift apart silently.

A failed load leaves the card as it found it.

Admission in the batched engine

The batched engine extends the same discipline to each request: admission reserves cache pages for the prompt and the maximum output before a sequence is admitted, and refuses with a typed reason when the pages are not there. The daemon's contract in Every request accounted for rests on that refusal being exact.

Pages free for the requestadmitted
Pages shortrefused, typed reason
prompt
maximum output
shortfall
Illustrative page counts.

What this demonstrates

Memory discipline on shared hardware: a failure path that restores the state it found, an admission check that is proven to match the allocation it predicts, and the overhead the driver adds counted rather than assumed away.

The crash-safe rule and the mirrored arithmetic were my requirements; AI coding agents, a mix of frontier and open-source models, implemented and tested them under my governance system.