Corey James- 50.001/sbuffers per second across the soak, against 20 ms pacing
- 250 msthe single retry in a forced mid-call reconnect
- 67 msround-trip media latency at most in a single probe, 65 at least
I rang a Microsoft Teams number, a digital employee answered, and we had a two-way spoken conversation. The speech recognition, the conversation engine and the voice all ran on GPUs in my own rack. Nothing about the call touched a third-party AI cloud. Making that possible meant building a bridge between two worlds that were never designed to meet: Microsoft's Windows-only real-time media stack and a Linux voice engine, joined by a purpose-built audio link that survived a thirty-minute soak and a forced mid-call reconnect before I would call it done.
The challenge
Microsoft Teams will hand raw call audio to a bot, and only to a Windows application built on Microsoft's real-time media SDK. My voice engine, with its speech models and conversation logic, runs on Linux GPUs, and I was not going to send call audio to someone else's AI service to avoid the mismatch. The bot had to sit on Windows in Azure, the brain had to stay on my hardware, and the two had to exchange audio in real time across clouds without the conversation noticing.
Constraints
A preview SDK with strict rules: unmanaged audio buffers with exact lifetimes and 20-millisecond pacing. Miss the pacing and the call stutters; mishandle a buffer and the process dies. Cross-cloud latency within the budget a conversation tolerates. No intelligence in the bridge, so it could be replaced without touching the AI. And a real identity: the bot joins calls as a governed digital employee in the Microsoft tenant.
My approach
I treated Teams as one more microphone and speaker. The engine already had a sealed audio seam, an abstract source and sink that its local voice interface used. Teams became another implementation of that seam. That reduced the problem to three parts: a Windows service that speaks the SDK's language and nothing else, an audio link with exactly the properties a conversation needs, and a Rust endpoint on my side that presents the link to the engine as a source and sink it already understood.
What I built
A .NET 8 Windows service on Microsoft's media SDK that joins calls, receives and sends 20-millisecond audio frames, and does nothing else.
A binary audio and video wire protocol over one authenticated WebSocket per call, implemented twice, in C# and in Rust, with the two implementations proven byte-identical against golden test vectors before either was trusted.
encodes each 20 ms frame for the link
decodes and encodes the same layout
A Rust endpoint that implements the engine's audio source and sink, so the engine cannot tell a Teams call from a local microphone.
Same-call reconnect: if the link breaks mid-call, the bridge reconnects within the same call and resumes without anyone redialling.
Deployment on an Azure virtual machine with managed identity and vault-held secrets, reachable only through an authenticated access layer.
Difficult decisions
Keeping the brain at home. The cheaper design was to let the Windows bot talk to a hosted speech service directly. That would have removed the bridge and put every call's audio in a third party's hands. I took the cross-OS bridge as the price of ownership.
Dropping audio on reconnect. When the link recovers, there is a choice between replaying the microphone audio captured during the break and discarding it. I drop it, and the person repeats themselves if they need to.
Replaying produces a colleague who answers a question from twenty seconds ago.
Trusting Teams for voice isolation. The local engine has its own filter for keeping one speaker in focus. In a Teams call the platform already isolates voices well, so I rely on that, and I wrote down the reasoning that would reverse it.
Results
The first call's spoken replies arrived several seconds late because speech synthesis was sharing a GPU with everything else; speech later moved to a dedicated host. Interruption handling on the Teams side, where a participant talks over the agent and it stops, came after that baseline.
Technologies used
C# and .NET 8 · Microsoft Graph Communications real-time media SDK · Azure virtual machines, Key Vault and managed identity · Rust with Tokio and Axum · Custom binary wire protocols over WebSocket · Real-time audio framing and pacing · Speech recognition · Neural speech synthesis
What this demonstrates
I integrate deeply with a vendor's real-time platform while keeping the parts that matter on hardware I control. I test failure before I test success: the soak and the forced reconnect were acceptance criteria, and the trade-offs are written down so the next engineer knows why the audio is dropped and when that should change.
I designed the bridge and the audio seam, set the acceptance tests, and directed AI coding agents, a mix of frontier and open-source models, under my governance system. Microsoft's media SDK and Teams are third-party; the bridge, the protocol and the engine integration are mine.