Engineering story · Helix

Fast transcripts that were wrong

GPU speech recognition that was far faster than the CPU path and quietly dropped sentences, and the correctness gate every GPU integration in the voice engine now passes

GPU speech recognitionDocker base imagesCorrectness fixtures
What was said What the fast GPU path returned · no error, no warning empty slots are whole sentences that silently vanished
The fixCUDA 12.8 base image → CUDA 13same recogniser, model and server code
1 linechanged: the base image
  • 25×faster than the CPU path at best on the GPU, 18× at worst
  • 4workarounds in the tensor library, none of which held

The voice engine recognises speech one utterance at a time: the voice-activity detector decides where a turn ends, and the whole turn goes to the recogniser in one call. On the CPU that call had a floor of about a second, because the recogniser pads every input to a thirty-second window before it listens. A second of dead air after every sentence is the difference between a colleague and a voicemail, so the recogniser moved to a server on a Blackwell GPU, and turns came back in 38 to 74 milliseconds.

One turn, by path · in the order the engine tries them
Native GPU server38–74 msFallback runtime235 msCPU, last resort~1 s
The CPU floor comes from padding every turn to a thirty-second window; the engine health-gates both servers before it starts a conversation.

What the fast path did

The transcripts looked fine in the log and were wrong in the room. Whole sentences were missing. The server had been built on a CUDA 12.8 base image, and on the consumer Blackwell architecture that toolchain ran the recogniser's kernels fast and incorrectly, with no error, no warning and no crash. The fix was to change one line, the base image, and the dropped sentences came back.

Workaround 1
Workaround 2
Workaround 3
Workaround 4
Base image changed
transcribed
a sentence that vanished
Gap positions are illustrative. Same recogniser, model and server code throughout.

What stayed in place

A second server, the same model under a different runtime, was built as a fallback. It exists so that a toolchain problem on the fast path has somewhere to fail over to. The CPU path stays as the last resort.

The rule it produced

Every GPU integration in the voice engine now gates on an output-correctness fixture before its speed means anything. The recogniser's existing defences against inventing words would not have caught sentences that simply vanished; only a fixture compares what came in with what should come out.

An invented word
Expected
Returned
caught by the existing defences
A missing sentence
Expected
Returned
caught only by the correctness fixture
The fixture is a known recording with a known transcript, checked before speed means anything.

A system can be fast and green and wrong.

What this demonstrates

The same lesson as Tests that passed without a GPU, in a different stack: a correctness fixture at the boundary is the only check that notices. It is also why the GPU pages on this site say which toolchain each result was proven on.

The decisions to gate on correctness and to keep a fallback were mine; AI coding agents, a mix of frontier and open-source models, built the servers and the fixtures under my governance system.