Engineering story · Inference Fabric

The queue that should not exist

I built an inference queue manager with two-stage backpressure and permit-carrying requests, then found the runtime already had a token-based queue, and reduced mine to a wrapper

RustTensorRT-LLM runtimeNVIDIA GPUs
Request level · reduced to a wrapper
permit · identity · release on every path
Token level · the queue that stayed
  • 64requests in flight at most, bounded by a channel
  • ~1,200lines in the wrapper that remains
  • 1,600lines in the token queue it almost duplicated

Between the services that need text generated and the engine that generates it, I built a queue manager. It was a good one. Then I read the runtime underneath it properly.

In-flight channel · each request carries its permit

The permit is taken before sending, so a waiting executor never stalls a submission.

Capacity checkAdvisory, never a gatea hard gate would serialise every request
IdentityFrom the engine itselfa caller naming the wrong model is refused at construction
Every way out of a request
Completion
Timeout
timeout
Rejection
reject
Shutdown
stop

Each ring is the permit going back to the channel; a safety timeout guarantees it, so the in-flight limit cannot leak.

What was already there

The saturated inference runtime already had a token-capacity queue: a lock-free multi-producer queue that reserved token budget with an atomic compare-and-swap, drained up to the engine's batch token limit, and returned a typed backpressure error when full. It did at the token level, where the engine actually has limits, what my manager did at the request level.

Two gates · requests over tokens request gate full
token budget starved below the batch token limit
One gate · a wrapper over the token queue
Request interfacepermit, identity, release on every path
admission delegated
token budget drained up to the batch token limit
The orange mark is the engine's batch token limit, the only limit the engine actually has.

Two layers of backpressure with different units would have fought each other, and the request-level one would have won by starving the token-level one of work.

The decision

The planning notes state the discovery and the resolution: the queue manager is reduced to a thin wrapper that presents the request-level interface the services want and delegates admission to the token queue. The permit, the identity check and the release-on-every-path discipline stayed, because they are properties of the interface rather than of the queue. The second queue went.

What this demonstrates

The willingness to delete a layer I had just built, and the habit of reading the runtime before adding to it. The wrapper and the queue it almost duplicated do different jobs now.

The decision to collapse the layer was mine; AI coding agents, a mix of frontier and open-source models, built both under my governance system.