Server-side echo cancellation on PSTN audio — advice, delay limits, and recommended config?

Self Hosting
, ,
1

Context up front: we’re not on LiveKit Cloud or rooms — we use the livekit Python package standalone for its AudioProcessingModule, doing server-side echo cancellation in a phone-bot pipeline (Twilio Media Streams, PSTN callers, 8 kHz).

Goal: cancel the caller’s speakerphone echo of our bot’s TTS server-side, before VAD/STT — PSTN callers have no client-side WebRTC AEC, and the leaked echo gets transcribed and triggers barge-in, so the bot interrupts itself.

What happens: APM works well when the echo path is short. But in our setup the echo returns 1–2 s (variable) after we feed the reference — outbound audio is buffered by Twilio and the carrier — and AEC3 then fails silently: ~0 dB reduction, no error. set_stream_delay_ms shifts the covered band but can’t span a 1–2 s variable delay.

Environment/versions: livekit 1.1.13 (pip), Python 3.12, Linux x86-64, 10 ms int16 frames at 8 kHz both streams. Usage skeleton:

apm = rtc.AudioProcessingModule(echo_cancellation=True, noise_suppression=False,
                                high_pass_filter=False, auto_gain_control=False)
apm.set_stream_delay_ms(100)
# on each outbound 10ms frame (as written to Twilio):
apm.process_reverse_stream(ref_frame)
# on each inbound 10ms frame (from Twilio):
apm.process_stream(in_frame)

Questions:

  1. Does anyone have tips for better usage of AEC3 for this use case?
  2. Is there an altogether different echo cancellation solution you’d suggest?
  3. Is there any way to give APM a large or dynamic delay hint? Is silent failure beyond ~500 ms the expected AEC3 behavior, and is there a recommended configuration for long, variable echo paths?
  4. How does LiveKit’s own SIP/PSTN bridge handle far-end device echo — server-side AEC3, Krisp, or is this handled at a different layer entirely?
  5. In your experience, is nonlinear residue from the far-end device’s own echo suppressor something APM can meaningfully cancel, or is the accepted answer “you can’t linearly cancel it — suppress or filter downstream instead”?
2

You are correct on #5 - . Phone hardware, cell network vocoders (AMR-NB / AMR-WB), and speaker clipping introduce heavy non-linear distortion. Linear AEC algorithms (like APM’s filter) cannot mathematically cancel non-linear speaker distortion—it must be suppressed non-linearly downstream using neural noise suppression or spectral subtraction.

Fix ideas that may help you:

  1. Dynamic Delay Buffer / Cross-Correlation Alignment: If they must use linear AEC3, they need to implement an external ring buffer using normalized cross-correlation (GCC-PHAT) between outbound TTS PCM and inbound Twilio PCM to estimate the dynamic offset $D$, and pass set_stream_delay_ms(D) dynamically. However, WebRTC APM will still struggle if $D > 500 \text{ ms}$.
  2. Deep Learning Voice Isolation (The Modern Fix): Replace or supplement APM with a model like Krisp BVC (Background Voice Cancellation) or ai-coustics.
  3. Mute/Gate during TTS Playback: For phone bots, the simplest architectural fix is Software Full-Duplex Suppression: duck or mute the STT/VAD engine while the bot is actively playing audio to Twilio, or use a hybrid approach where VAD requires high-confidence human voice triggers while TTS is active.

How LiveKit Handles Far-End Telephony Echo from my Understanding, and things I have noticed

  • LiveKit’s native SIP stack does not rely on standalone WebRTC AEC3 on long-delay PSTN pipes for this exact reason.

  • Instead, LiveKit relies on Krisp / Voice Isolation Models (e.g., Krisp BVC Telephony or ai-coustics) running directly on the agent audio pipeline or SIP trunk.

  • Deep-learning voice isolation models classify “the caller’s voice” vs. “background noise/leaked synthetic TTS”. They attenuate the leaked speakerphone audio downstream using spectral masking rather than trying to perform linear adaptive subtraction across a 2-second moving buffer.

Best of Luck, and happy coding! There is nothing better than seeing it in action, once you have finished. I spent a month just testing my app, before actually putting it to use. Then, I have the issue of wanting to add something else. " Well it works, but what will make it better" A never ending question, and the fun is over, once it works.