Hidden-state extraction

One forward pass, no generation

Inference Fabric was built for a distillation workload that needs a model's internal representation of every example, at dataset scale. Serving stacks return tokens, and running full generation to read one vector wastes the GPU. I added an extraction mode to the runtime itself, so the GPU spends nothing on tokens nobody needs.

An engine built without a key-value cache routes to a raw TensorRT path that runs one forward pass per batch and returns the final hidden state of each sequence. A consumer has to pass a contract check before it may join the broker.

Generate to read one vector

forward, sample, forward, sample: tokens nobody needs

Extract

one probe vector, nothing generated

Buffers bound once, batches in a fixed order

At load the runtime enumerates every tensor the engine exposes. After that, a batch packs tokens contiguously, writes positions and request types, and runs a strict order. Re-binding after every shape change is what keeps TensorRT from reading stale addresses.

Allocated once, at maximum shape

Device

per token, packedper sequencestaticoutput

Host

per sequencestatic

Every batch, on one stream

set every dynamic shapere-bind every addresscopy device-side inputsone forward passone copy-out

Asking the engine which token to gather

The hard part is one integer per sequence. A wrong guess at it ruins the dataset quietly, on every probe. So the runtime measures the engine's gather convention at start-up and fails closed on an ambiguous answer.

Validating vectors later cannot catch a well-formed vector for the wrong token.

One tokenTwo tokens Convention A

Convention B

Output fingerprints, FP16 decoded by hand. Here A gives two distinct, non-trivial vectors and B gives the same one twice, so A is frozen and one function computes every gather index from it.

Refuses to start when every output is zero, when both conventions agree, or when the outputs are inconsistent.

When both conventions look consistent, the detector takes the one with the larger separation between sequences. That is the method's weakest point, and I keep it visible in the code.

Results that describe themselves

Results leave as self-describing payloads carrying data type, shape, a hash and the identity of the extraction request, so a stored vector can be interpreted without knowing which engine produced it. The vector store checks lineage back to the originating work item before it stores anything. The throughput side of the same runtime is From 44K to 163K tokens per second, and the workload behind it is The workload that shaped the platform.

One run through the broker · tokens per second

Average25,800Peak45,500

I can see a capability mainstream serving stacks do not offer, design it from producer to storage, and find the one silent failure that would corrupt the result. The forward pass is NVIDIA TensorRT, with no custom model kernels.