Corey James- 91,643call buffers moved in a 30-minute soak
- 50.001/sbuffer rate held across that soak
- 0receive stalls in the same run
- 30 msdenoiser latency gate at the 95th percentile
One audio seam for a desk and a Teams call
In Helix I wanted one voice engine to work on a desk microphone and inside a Teams call without the conversation logic knowing which. I gave the Rust engine one sealed audio seam: an abstract source and sink with one frame contract, so a new audio endpoint is an implementation and the engine does not change. The same turn-taking and interruption behaviour runs in both.
A front end of composable stages
The front end is a chain of stages over that frame contract, each reading and writing the same frames. Echo cancellation uses the WebRTC AEC3 algorithm with the engine's own playback as reference, in ten-millisecond blocks with the delay seeded from device clocks, so the agent does not hear itself; in a Teams call the reference is the engine's playback into the call. A low-latency denoiser, vendored and patched to build on stable Rust, removes noise on the CPU with its attenuation capped, under a latency gate. An optional operator voiceprint filter on speaker embeddings, enrolled without keeping any recording, silences other voices before voice-activity detection, so a television cannot interrupt the agent or trigger a tool. A voice-activity model then marks where speech is.
The voiceprint filter is an input-quality stage for the desk microphone only; Teams calls rely on Microsoft's voice isolation at the capture device, for written reasons. When the recogniser's confidence is low, the turn comes back empty.
An agent acting on words nobody said is worse than one that asks again.
Stopping the moment a person speaks
What a person notices is that they can talk over the agent and it stops, with no stale audio played afterwards. Underneath, the real-time callbacks copy and return and do no allocation, locking or logging: the capture callback downmixes and pushes into a lock-free ring of about 200 milliseconds, and a dedicated sender owns pacing.
Coloured: the current turn. Grey: blocks from the turn the person just interrupted, which play as silence.
At most one device period, already latched in the hardware, can still sound.
The interruption behaviour is in Interruptions that only people can make; the echo-cancellation proof in A proof that proved nothing.
The Teams link
The Teams side has the hardest real-time constraint I have worked to. Microsoft's real-time media SDK delivers call audio only to a Windows application, in 20-millisecond buffers with strict lifetimes. My .NET service copies each buffer and returns at once, and a paced sender keeps the call from stuttering. Audio crosses from an Azure virtual machine to my Linux host over a binary protocol on one authenticated WebSocket per call, written in C# and in Rust and proven byte-identical against golden test vectors. See A self-hosted colleague in a Teams call.
- C# encoder and Rust decoderbyte-identical
Dropping what was captured during the break means the agent never answers a question from twenty seconds ago.
Cutting the felt gap
Latency work followed the same path. Timing each turn exposed a fixed overhead in the speech-synthesis server; streaming synthesis with cached voice conditioning removed it, and speech moved to a dedicated GPU host: From two seconds to under one.
The first word took the same time for a short clause or a long one; the constant was the synthesis server, not the speech.
I can engineer real-time audio across operating systems and clouds with the discipline the medium demands, and keep the conversation logic independent of any one platform. Microsoft's SDK and the open speech and audio models are third-party.