Capability

Voice agents in calls and meetings

Speech recognition, synthesis and turn-taking on my own hardware, joined to the platforms people already use

Rust voice engineSpeech models on my own GPUsTeams media bridge
Full duplex · it listens while it speaks Recognition, conversation and voice on my own GPUs · nothing stale plays after a stop
  • 25×faster speech recognition on the GPU than on the CPU path, 18× at the least, in real turns
  • < 1 msto play the lines an assistant says most, pre-rendered per voice
  • 5,668turns in a ten-minute loopback soak, every one processed

An AI that people can talk to in a real call has to answer in conversational time, stop when it is interrupted, and keep the audio away from third-party AI services.

I built a conversational voice engine in Rust with speech recognition, neural speech synthesis, floor control so several AI voices share one conversation without talking over each other, interruption handling so a person speaking stops the agent, and tool calls made mid-conversation. The gap between the end of a question and the first sound of the reply was taken apart stage by stage. See From two seconds to under one.

Question to first sound · drawn to one scale Before · about 2.1 s silence 700 ms · recognition on the CPU, about 1 s · first voice frame about 310 ms After · about 0.9 s silence 500 ms · recognition 38 to 74 ms on a GPU · first voice frame about 300 ms

One clock now decides the person has finished; recognition moved to a GPU behind an output-correctness fixture; the first sentence is short by contract because first audio waits on it; and the voice streams from a model that came with no streaming.

The unlabelled sliver after the voice is reasoning and the first sentence. Nothing about the reply got shorter.

What counts as an interruption was measured rather than guessed: the engine listens while it speaks, a person can talk over it and it stops at once, and its own echo cannot stop it. See Interruptions that only people can make and A proof that proved nothing.

Who may take the floor
  • A person speakingalways wins
  • The first word of an interruptionis heard
  • A false stopre-speaks
  • The agent's own echonever

Echo and room noise return as brief, low-confidence bursts; a person does not. The threshold sits between the two, set from real sessions.

Echo cancellation score, echo-only recording
The real reference36.8 dBAn unrelated reference37.8 dB

An excellent number that discriminated nothing: the suppressor did all the work. The rebuilt proof carries a negative control.

A proof that passes with the wrong input is a proof of the test harness.

I joined the engine to Microsoft Teams through a purpose-built real-time media bridge with byte-identical implementations on both sides, soak-tested for thirty minutes and through a forced mid-call reconnect. When a Teams turn ran slow, the cloud hop was the obvious suspect and measurement cleared it. Start with A self-hosted colleague in a Teams call and One percent network, seventy-eight percent GPU.

A slow Teams turn, about six seconds, taken apart
the network, about 1%
a speech GPU shared with a busy embedding service, about 78%
everything else

The Azure-to-home round trip measured 65 ms. Speech synthesis moved to a dedicated GPU host; the cloud bridge stayed exactly as it was.

On a desk microphone an optional voiceprint, enrolled from eight guided samples without keeping a recording, lets the engine listen only to its operator; it is deliberately not used on Teams, where the platform already isolates voices. A deterministic layer turns written answers into speakable ones while the screen keeps the written form, and no real human voice is cloned without documented consent. See Listen only to me and Saying numbers out loud.