Technology · Discipline

GPU admission control and co-residency

Two models on one GPU, with the second admitting work against measured memory and refusing with a number: a 0.6B router beside a 30B model, through a 2,048-client burst.

0.6B router beside a 30B modelCapabilities model · Inference Fabric · HelixBlackwell GPUs
One Blackwell GPU · two models
30B model, resident and serving0.6B router
Admitted against measured memory
busyshortfall and retry hint;
the request never starts
2,048clients in one burst, none failed, rejected or corrupted
  • 2,740MiB, the router's whole footprint beside the resident model
  • 85.9/srequests served through the 2,048-client burst
  • 45.5/sshort requests sustained

A router beside a 30B model

Capabilities model had to run on a Blackwell GPU whose memory belonged to a 30-billion-parameter model that stayed resident and serving. Agents call the discovery model before most actions, so many would call it at once, and a wrong answer under load would be worse than a slow one.

I gave it a fixed memory budget and made two thousand concurrent clients an acceptance criterion from the start.

I set the acceptance numbers before the engine exists so the result cannot be reframed afterwards.

The engine admits work against the memory it has measured, so overload becomes a queue and then a refusal and never a corrupted answer. Before it accepts a request it checks that the request's key-value memory fits; when it does not, the caller gets a typed busy response carrying the shortfall and a retry hint, and the request never starts. The engine binds into the serving path only after a canary request succeeds, and it drains gracefully. A load ladder climbed from one client to 2,048 with abort conditions at every rung, so a failure stopped the test. The full account is A router beside a production model.

On mixed traffic the engine fell far short of its pre-registered target, because long requests needed more key-value memory than was left beside the resident model, and I reported the honest figure.

Mixed traffic · requests per second
Target, set first200Flattering count~14Reported6.9
Key-value memory for one long request
Needed~6 GBAvailable~1.4 GB
The memory gap explains the throughput gap.

A GPU that never idles and never stalls

Inference Fabric had the opposite problem. I put admission control in front of TensorRT-LLM's in-flight batcher, and backpressure counts consumption at the GPU, which keeps the broker from overfeeding a stalled engine.

Retry buffer, first
Broker pulls, bounded
One step
decode rows firstminimum batch tokensprefill, then free slots
Illustrative proportions. Slot gating uses hysteresis so completions cannot collapse, and token budget goes back to the broker only once a batch reaches the GPU.

An early version of the runtime stopped the GPU entirely with sequences in flight; I added a progress-only step that always advances active sequences. See From 44K to 163K tokens per second.

Shown free before a render starts

Helix shares GPUs between real-time voice and the agents' render jobs, and I decided a render may start only after the service has shown a GPU is free. A GPU that looks idle has to be shown idle, and missing telemetry fails safe.

Before every start
A render that starts
Telemetry missing
The first start
placed by device UUIDno compute processcompute budget
After the stopGPU memory: 0 MiB
The first start was refused because a monitoring process held every GPU; that one observer got an explicit exception.

I protect correctness ahead of throughput when two workloads share a card. These are single-host designs; co-residency is managed in software with budgets and admission.