Engineering story · Helix

A proof that proved nothing

An echo-cancellation score looked excellent until an unrelated reference scored higher; the rebuilt proof uses a negative control, and the same discipline runs through the voice engine's tests

Echo cancellationNegative controlsRust audio tests
Echo-only recording · echo return loss enhancement
The wrong input scored higher
37.8 dBfor audio that had nothing to do with the echo
  • 5 of 5content words recovered from synthesised speech
  • 0for a pure tone, which came back as a description of music
  • 2layers of echo cancellation, the engine and the desk device

The voice engine cancels its own echo with the WebRTC AEC3 algorithm, fed the engine's own playback as the reference, in ten-millisecond blocks, with the delay seeded from the device clocks. The first proof that it worked reported an echo return loss enhancement of 36.8 decibels on an echo-only recording, which is an excellent number. Then the same test was run with an uncorrelated reference, audio that had nothing to do with the echo, and it scored 37.8.

What the number was measuring

The canceller has a suppressor stage that gates low-level signal, and on an echo-only recording the suppressor was doing all the work: whatever the reference, it attenuated the echo because the echo was all there was. The score was real and it discriminated nothing.

A proof that passes with the wrong input is a proof of the test harness.

The rebuilt proof

The replacement test is double-talk, a person speaking over the agent's playback, which is the case echo cancellation exists for.

Double-talk · near-end speech preserved
the right reference0.97the wrong reference0.01

The negative control is the proof: the number moves when the input is right and collapses when it is wrong, so it measures the canceller and not the suppressor. The canceller's binding to its volatile native library is isolated in one file behind a trait.

The same discipline elsewhere

Synthesised speech
content words recovered
A pure tone
none: it came back as a description of music
  • A corrupt detector model, suite passes by skippingbefore
  • A corrupt detector model fails loudlynow
  • Stub buildssearched for in the code

A mishearing in one run was kept as stronger evidence of a real pipeline than a perfect match would have been.

What this demonstrates

I test audio by asking whether a passing number could pass for the wrong reason, and by building the control that would catch it. The ERLE figure was thrown out, and the engine's acceptance tests are the ones that fail when they should.

The decision to discard the first proof was mine; AI coding agents, a mix of frontier and open-source models, rebuilt the tests under my governance system.