Engineering story · Capabilities model

Trying the industrial path first

Evaluating the standard vendor serving stack for a custom decoder, and closing that path on measurements

3 min readCapabilities modelTensorRT-LLMGrammar-constrained decoding
Two candidate engines · one kept Vendor industrial stack · traced, probed, converted, measuredCustom decoder · same answers, no repeats, catalog names only
each orange mark is a finding: grammar, conversion, throughput
3findings that closed the vendor path
  • 4checks on the vendor path, any one of which could have kept it
  • 3guarantees any engine had to keep: identical answers, no repeats, catalog names only
  • 1,000×weight corruption on some tensors after the vendor conversion

When the tool-discovery model needed to serve two thousand concurrent clients, the obvious answer was the vendor's industrial serving stack. I directed that it be tried first, before any custom engine was justified. It was traced to its extension point in source, probed against my decoding constraints, and measured in the vendor's own container. It did not fit, for three reasons I could show, and the custom engine was kept.

The challenge

Building a custom GPU decoder is expensive and easy to justify badly.

If a standard stack would serve the model at the required concurrency with the required guarantees, the custom engine was the wrong choice however satisfying it was to build.

Constraints

Identical answers, so switching engines could not change what an agent was told. The decoding constraints are what make the model safe to use. And a shared GPU: the stack had to fit beside the 30B model.

My approach

Trace, probe, measure. Find where the vendor stack allows a decoding constraint to be inserted, read that code, probe it with the real constraint, convert the model with the vendor's tooling and check the weights, then run a throughput ladder in the vendor's container with and without the 30B model resident.

The vendor path, checked in order
ConstraintNo-repeat cannot be enforced soundlythe grammar is layered in a way that lets repeats through
ConversionWeights silently corruptedon some tensors, by orders of magnitude
ThroughputToo little beside the modelthe memory that fits leaves a fraction of the ceiling
KeptThe custom enginejustified by three findings anyone can reproduce

What I built

Probes against the stack's guided-decoding extension point using the real catalog constraint.

A conversion verdict comparing converted weights against the source.

A throughput ceiling study run inside the vendor container, with the card to itself and with the 30B model resident.

Difficult decisions

Closing a path I had chosen. I had directed the vendor-first approach and expected it to work. Three findings closed it, and the throughput one is drawn below.

Results

Measured in the vendor's own container, on the shared GPU:

Requests per second, vendor stack in FP8
Card to itself492Beside the 30B model56
The ceiling needs the whole card; the card is not free, so the ceiling never applies.

Technologies used

TensorRT-LLM · Grammar-constrained decoding libraries · CUDA · NVIDIA Blackwell GPUs

What this demonstrates

Vendor-neutral judgment. I try the standard tool first, and when it does not fit I can show exactly why with measurements, which is the only justification a custom engine deserves.