Engineering story · Helix

From two seconds to under one

Question to first sound, taken apart stage by stage: the silence window, recognition, reasoning, the first spoken sentence and synthesis, each cut named and each benchmark row carrying its own definition

Rust voice engineGPU speech recognitionStreaming synthesis
Question to first sound · before After
silence window recognition first words first audio frame time no longer spent
0.9 squestion to first sound, down from about 2.1
  • 400 mscut at most on complex answers by keeping the first sentence short, 300 at least
  • 5,668turns in a ten-minute loopback soak, every one processed
  • 96 KBmemory growth across that soak

A spoken assistant is judged by one interval: the gap between the moment a person stops talking and the moment they hear a reply begin. In the voice engine's first working version that gap was more than twice what it is now.

The stages

Before the work, the chain was roughly: 700 milliseconds of silence before the engine decided the person had finished, about a second for speech recognition on the CPU, a few tens of milliseconds for the reasoning model's first words, and about 310 milliseconds for the synthesiser's first frame of audio, with the first sentence having to complete before it could be spoken.

The silence window went from 700 milliseconds to 500, after an earlier fix had found two silence timers nested so the engine waited for one and then the other.

How long the engine waits before deciding a person has finished · ms
Two nested timers≈1,020One clock from the last voiced window704 ± 16Silence window now500
Replacing the nested timers with one clock reclaimed 320 ms before the window itself was shortened.

Recognition moved to a GPU and dropped from about a second to between 38 and 74 milliseconds, with the correctness catch told in Fast transcripts that were wrong. The speech contract was given a rule that the first sentence is short and direct, because first audio waits on the first sentence. The synthesiser was taught to stream, in Building the streaming my speech model did not ship, and the lines an assistant says most often were pre-rendered, in Three hundred milliseconds of hello.

Nothing about the reply got shorter. The gap was taken apart into its stages.

Starting before the person has finished

Two mechanisms work on the gap from the other side. The engine begins preparing an answer after a brief pause rather than waiting for the full silence window, and silently discards that preparation if the person resumes, so a hesitating speaker is never cut off and a finished one is answered sooner. And two answers are generated together: a short opener that needs no tools, and the detailed answer, joined into one stream with any restatement dropped, so a person hears something within the first few hundred milliseconds while the detail is still being produced. An earlier design speculated on the audio itself rather than the text, and two synthesis streams sharing one synthesiser starved the active one into chopped audio; speculation is now text only, replayed into the unchanged pipeline when the turn is granted.

Two answers, one stream
any restatement of the opener is dropped at the join
A preparation started in a pause is thrown away silently if the person keeps talking.

A number must name its row

Every figure in the engine's benchmark file carries its definition: what was measured from which event to which, on which path, warm or cold.

Benchmark bands · milliseconds, on a 0 to 350 scale
Recognition, conversational
26 to 74
Recognition, long instruction
96 to 145
Reasoning, first delta
25 to 66
Synthesis, first frame
300 to 350
Long answer, 18 clauses
304 to 327
Synthesis first frame: mean near 325, ninety-fifth percentile near 343. The long answer ran two minutes and forty-two seconds, first frame per clause.

What this demonstrates

I do latency work as decomposition: the gap named stage by stage, each cut made where it belonged, the stages that overlap made to overlap, and every number tied to the row that defines it.

The decomposition and the speech-contract rule were mine; AI coding agents, a mix of frontier and open-source models, implemented and measured each stage under my governance system.