Corey James- 2.7 GiBthe engine's whole footprint beside a 30B model
- 85.9/srequests served during the largest burst
- 0answers failed, rejected or contaminated in that burst
The tool-discovery model had to live on a GPU whose memory belonged to a 30-billion-parameter serving model. I set two thousand concurrent clients as an acceptance requirement, and built an engine that admits work against measured memory and refuses with a number when it cannot. The sustained throughput target I had pre-registered for mixed traffic was missed by a wide margin, for a reason I can name, and that miss is on this page.
The challenge
A discovery model that agents call before most actions will be called by many agents at once. If it falls over under load, every agent stalls at the same moment. It had to serve a burst of thousands, never return a corrupted answer under pressure, and do that inside whatever memory the 30B model left behind.
Constraints
A hard memory budget on a shared Blackwell GPU, with the 30B model resident and serving. Correctness under overload came before throughput. A pre-registered target for sustained mixed traffic, written down before the engine existed.
My approach
Admit before you load. The engine measures the memory it has, decides whether a request fits before accepting it, and reports busy as a typed refusal with a number attached. Concurrency of two thousand was an acceptance criterion from the start, tested by a load ladder from one client to 2,048 with abort conditions at each rung.
The engine joins the serving path only after a canary request succeeds.
What I built
A batched, continuous-decode engine with paged key-value memory, so many clients share the GPU without each reserving a full context.
A load ladder from one to 2,048 clients with abort conditions, so a failure stops the test instead of being averaged into it.
Difficult decisions
Reporting the miss. On mixed traffic the engine fell far short of the target, and a friendlier accounting method would have shown about 14 requests per second. I reported the plain number with its cause, drawn below.
The miss is structural, and saying so is more useful to the next design than a better-looking number.
Results
On one GPU with the 30B model resident:
Long requests cannot fit beside the 30B model, so mixed traffic waits on them.
Technologies used
CUDA · Rust · Unix-socket wire protocol · Model Context Protocol over TLS · Memory-aware admission control · NVIDIA Blackwell GPUs
What this demonstrates
Systems engineering under a hard resource limit, with correctness protected ahead of throughput, and capacity reporting that tells the reader what the system cannot do as plainly as what it can.