Engineering story · Inference Fabric

A second engine behind the same door

The vendor's executor could not run engines with no key-value cache or an extra output, so a raw-TensorRT backend went in behind the same C interface, and the Rust side never knew

C++ backendInference FabricRaw TensorRT runtime
The same calls from above
0changes above the C interface
  • 8 KBcopied back to the host per sequence
  • 128tokens into a mining engine, for one forward pass
  • 6 lineschanged in the vendor's model graph to gather the output to the last token

Hidden-state extraction needs an engine that generates nothing. I built those engines, in Engines for the card you have, and the vendor's executor refused to run them.

A generation engineruns
Mining engine: no key-value cachepaged cache asserted
Mining engine: an extra outputnot on the allow-list
The extra output is the representation before the output head; with nothing to decode, the engine has no cache.

The constraint I set

The runtime has a Rust half that talks to the broker and a C++ half that talks to the GPU, joined by a C interface.

I did not want the Rust half, the provisioner or the broker to know that mining engines were different.

So the rule for the second backend was no new configuration surface, no Rust-visible behaviour, and the engine's own tensor enumeration as the only source of truth about what it wants.

The backend

The C++ library detects the backend kind from the engine's configuration and, for a mining engine, loads it through the raw TensorRT runtime rather than the executor. All of the work happens once, at load.

Once, at load
List tensors
Check name, shape, type
Override host placement
Confirm no cache
Allocate once
Every batch
Bind inputs
Bind outputs
Launch
The engine's own tensor list is the only description of what it wants.

The output is already gathered to the last token inside the model graph.

Copied back to the host · batch of 2,000
Full-sequence output4 GBLast token only16 MB

What the first attempt taught

The earlier mining path lived in the Rust half, and it broke a rule I keep: one path.

First attempt: payload cloned per sequence13 MB × 1,600
Now: buffers allocated onceonce
The first attempt grew until the broker ran out of memory. It had also copied 300 lines of start-up code and returned before the metrics server started; the new backend has one path.

One contract survives from that work: an extraction failure never blocks completion, so a sequence whose extraction fails completes with no payload and the pipeline keeps moving.

What this demonstrates

Extending a vendor runtime at the layer where the extension is invisible to everything above it, with the engine itself as the authority on its own interface, and the discipline to delete a working first version that broke the architecture.

The backend boundary and its guardrails were my design; AI coding agents, a mix of frontier and open-source models, implemented it under my governance system.