A noise-cancellation FrameProcessor that resamples to its model’s rate and back must not force its output length to match the input frame. rtc.AudioResampler.push() returns variable-size chunks, so trimming the surplus and zero-padding the shortfall destroys ~16% of caller audio — continuously, not just at stream start — whenever the room rate isn’t 16 kHz and the frame size is 50 ms. That combination is AudioInputOptions’ default.
This is a bug we shipped ourselves, not a bug report about someone else. We are posting it because of where the pattern came from and where else it still is.
What happened, in order:
- Our agents were mis-hearing clearly-spoken words on phone calls — proper nouns, spelled-out letters — with nothing wrong
in any log. Swapping STT providers changed nothing, because the damage was upstream of STT. This went on for months. - We traced it to our own WebRTC-APM noise-cancellation processor and measured it destroying 16–22% of every caller’s
audio. That is the processor our agents actually run, and fixing it is what solved the problem. - Our processor’s docstring says “structure mirrors DTLN’s DTLNNoiseSuppressor” — because that is what we used as the
reference when writing it. So afterwards we went and looked at livekit-plugins-dtln itself. - It has the identical defect, measured at 16.03% on the same geometry. We do not use DTLN and never have; we only checked
it because our copy came from it.
So the actionable part for LiveKit is that the shipped plugin still carries this, and the shipped plugin is what people read when writing their own. We are the worked example of the pattern being copied verbatim — including the bug — by someone who never ran DTLN at all.
Everything below reproduces against livekit-plugins-dtln, since that is the copy you can act on. Our own numbers are cited where they add evidence.
Versions: livekit-plugins-dtln 0.1.5, livekit-agents 1.6.8, Python 3.12, Linux.
The code
livekit/plugins/dtln/noise_suppressor.py, lines 319–324:
Trim or pad to exactly match the input frame length
target = frame.samples_per_channel
if len(out_samples) > target:
out_samples = out_samples[:target]
elif len(out_samples) < target:
out_samples = np.pad(out_samples, (0, target - len(out_samples)))
The plugin downsamples to 16 kHz, runs DTLN, then upsamples back. Both resamplers are streaming and return whatever whole output frames are available for a given input chunk — that count is not required to match the input, and in general doesn’t.
Forcing it to match means:
- when the resampler returns more than target, the surplus is discarded — those samples are gone, they are not carried into the next call;
- when it returns less, the shortfall is zero-filled — digital silence is spliced into the middle of the caller’s speech.
Both happen continuously, not just at stream start.
Why this is worth fixing even though DTLN may have few users
The plugin reads as the reference implementation for a FrameProcessor-based noise canceller — lazy resamplers to and from the model’s fixed rate, a queue to absorb arbitrary frame sizes, a fixed-size block loop. It is the natural thing to model your own against, which is exactly what we did, trim/pad tail and all.
So the blast radius is not DTLN’s install count. It is everyone who has read this file while writing their own processor. We are one confirmed instance, and we never ran DTLN in production for a single call.
Two properties make it unusually good at hiding, and they applied to our copy just as much as to this one:
- It raises nothing. Output length always equals input length, every frame is a legal size, no exception, no warning. The only symptom is degraded transcription, which sends investigators to the STT vendor.
- It vanishes under test. Any harness at 16 kHz (where both resamplers are None and the branch is skipped) or at 10 ms frames measures 0%. Both of ours were built that way, so every noise-cancellation comparison we had ever run — months of them — was structurally blind to it.
Measurements:
Six seconds of tone-plus-noise through _process(), measuring how much of the output is digital silence that was not present in the input:
┌───────────┬────────────┬──────────┬───────────────────┐
│ room rate │ frame size │ strength │ silence in output │
├───────────┼────────────┼──────────┼───────────────────┤
│ 16000 │ 50 ms │ 0.3 │ 0.01 % │
├───────────┼────────────┼──────────┼───────────────────┤
│ 16000 │ 50 ms │ 1.0 │ 0.05 % │
├───────────┼────────────┼──────────┼───────────────────┤
│ 24000 │ 50 ms │ 0.3 │ 16.03 % │
├───────────┼────────────┼──────────┼───────────────────┤
│ 24000 │ 50 ms │ 1.0 │ 16.06 % │
├───────────┼────────────┼──────────┼───────────────────┤
│ 48000 │ 50 ms │ 1.0 │ 17.99 % │
├───────────┼────────────┼──────────┼───────────────────┤
│ 24000 │ 20 ms │ 1.0 │ 0.03 % │
├───────────┼────────────┼──────────┼───────────────────┤
│ 24000 │ 10 ms │ 1.0 │ 0.02 % │
└───────────┴────────────┴──────────┴───────────────────┘
Two things worth calling out:
- 24 kHz / 50 ms is the default. AudioInputOptions.sample_rate is 24000 and frame_size_ms is 50, so an agent that never touches these settings is in the affected configuration.
- It is invisible at 10 ms and 20 ms, and invisible at 16 kHz (where both resamplers are None and the whole branch is bypassed). Any offline harness built at 16 kHz or 10 ms frames measures a code path production never takes — ours did, for months, which is why we did not catch this earlier.
On telephony audio the practical effect is that STT mis-transcribes clearly-spoken words — proper nouns and spelled-out letters especially — with no error anywhere in the logs. It looks like an STT accuracy problem, so it gets chased in the wrong place.
Reproduction:
Self-contained, no audio files needed:
import numpy as np
from livekit import rtc
from livekit.plugins import dtln
def frames(sig, rate, ms):
n = rate * ms // 1000
pcm = (np.clip(sig, -1, 1) * 32767).astype(np.int16)
return [rtc.AudioFrame(data=pcm[i:i + n].tobytes(), sample_rate=rate,
num_channels=1, samples_per_channel=n)
for i in range(0, len(pcm) - n + 1, n)]
def run(rate, ms, strength, seconds=6):
rng = np.random.default_rng(4)
t = np.arange(rate * seconds) / rate
sig = (0.4 * np.sin(2 * np.pi * 220 * t) + 0.2 * np.sin(2 * np.pi * 440 * t)
- 0.05 * rng.standard_normal(len(t)))
nc = dtln.noise_suppression(strength=strength)
out = [np.frombuffer(nc._process(f).data, dtype=np.int16)
for f in frames(sig, rate, ms)]
o = np.concatenate(out).astype(np.float32) / 32768.0
return 100.0 * int(np.sum(np.abs(o) < 1e-6)) / len(o)
for rate, ms in ((16000, 50), (24000, 50), (24000, 10)):
print(rate, ms, f"{run(rate, ms, 1.0):.2f}% silence")
Suggested fix
Stop treating output length as a function of input length. Keep a persistent output buffer across calls, emit exactly one frame of the expected size, and carry the remainder forward:
- push the upsampled frames into a persistent queue rather than concatenating per-call;
- emit frame.samples_per_channel samples from the front of that queue and keep the rest for the next call;
- while the queue is still filling (first couple of frames), return the input frame unchanged rather than padding with zeros;
- clear the queue whenever the resamplers are rebuilt, so a mid-call track republish at a different rate cannot leak stale samples.
One constraint that is easy to miss: the emitted frame length cannot simply be “whatever the resampler produced”. If RoomInputOptions.auto_gain_control is on, LiveKit’s own rtc.AudioProcessingModule.process_stream() runs downstream and
requires whole 10 ms blocks. We measured 788 / 1080 / 1576-sample frames each producing a Rust panic in libwebrtc apm.rs - which is not a catchable Python exception, so it takes the worker process down rather than failing the one call. A fix must keep emitting a legal 10 ms multiple; variable-length output is not an option.
The cost of the buffered approach is a small, bounded priming delay at stream start (a fraction of one frame), and no added steady-state latency — the output frame is still produced synchronously on every input frame.
We built both plausible fixes and measured them-
Worth sharing, because the obvious cheap fix looks fine on a silence metric and still degrades the audio.
Since 10 ms and 20 ms frames are unaffected, an obvious workaround is to chop each incoming frame into 10 ms pieces and feed those through unchanged. We compared that against buffering properly, on identical input. “Fidelity” is the correlation of the output against the source at the correct lag — it cannot reach 1.0, because a suppressor is meant to alter the signal, but it separates “changed” from “mangled”:
┌──────────────────────────┬─────────┬────────┬──────────┐
│ approach │ silence │ length │ fidelity │
├──────────────────────────┼─────────┼────────┼──────────┤
│ plugin as-is │ 16.03 % │ 100 % │ 0.105 │
├──────────────────────────┼─────────┼────────┼──────────┤
│ chop into 10 ms frames │ 0.01 % │ 100 % │ 0.769 │
├──────────────────────────┼─────────┼────────┼──────────┤
│ buffer + carry remainder │ 0.00 % │ 100 % │ 0.915 │
└──────────────────────────┴─────────┴────────┴──────────┘
The chopping workaround removes the injected silence entirely, so a zeros-in-output metric declares it fixed — but the plugin is still running its own 24k→16k→24k round trip on every 10 ms piece, and the fidelity number shows the cost.
Buffering avoids that: the caller does one downsample and one upsample, and DTLN only ever sees 16 kHz, where its trim/pad branch is a no-op.
Verified across geometries after the change — 24k/50 ms 0.01 %/0.915, 24k/10 ms 0.01 %/0.913, 48k/50 ms 0.03 %/0.991, 16k/50 ms 0.01 %/0.911 — with zero non-10 ms-multiple frames emitted at any of them.
This is the fix we run in our own WebRTC-APM processor against real PSTN traffic: it measures 0 samples discarded and 0 padded over multi-minute calls, with no change in mouth-to-mouth latency, and the mis-transcription problem that started all of this is gone. We have applied the same pattern to DTLN behind a wrapper, but since we do not use DTLN in production those numbers are bench-only.
Thanks Troy.
Looking for a self-hosted livekit dashboard