STT: AssemblyAI using `@livekit/agents-plugin-assemblyai@1.5.3`
Turn detection: `new inference.TurnDetector()`
No agent-side noise-cancellation or voice-enhancement plugin is configured
I do not explicitly provide a VAD, so AgentSession automatically provisions the bundled inference VAD.
Results
Using the same caller utterance:
Test
Maximum probability
Detected
Full continuous call replay
0.1386
No
Fresh VAD with the same 32 ms alignment
0.4708
No
Fresh VAD with the boundary shifted by 8 ms
0.9785
Yes
The successful run produced 14 inference windows above the 0.5 threshold.
I also found:
Keeping the full call history and shifting the global window phase by 0–28 ms produced probabilities between 0.0437 and 0.4959. Every phase missed.
Resetting the VAD and creating a new VAD handle produced identical results.
Depending on the exact reset boundary, the same utterance scored between approximately 0.24 and 0.98.
Real-time playback produced the same results as faster-than-real-time replay.
Agents 1.5.3 and 1.5.5 behaved identically.
This looks like an interaction between retained VAD state and fixed, non-overlapping 512-sample window alignment.
Expected behavior
I would not expect a clearly audible utterance to change the VAD from approximately 0.24 to 0.98 based on only a few milliseconds of window alignment.
Actual behavior
The VAD sometimes scores the entire utterance below the default 0.5 threshold. No speech event is generated, so my agent remains silent while waiting for input.
Questions
Is this a known limitation or bug in @livekit/local-inference?
Which Silero model version is embedded in @livekit/local-inference@0.2.6?
Should the bundled VAD state be reset after completed caller turns?
Not a local-inference bug that’s Silero itself. It’s a stateful RNN over fixed 512-sample frames, so per-window probability is genuinely a function of both alignment and carried state. An 8 ms shift taking you 0.47 → 0.98 is expected for that model, not a defect.
Two things in your setup are likely making it worse. inference.VAD resamples everything to 16 kHz with AudioResamplerQuality.QUICK (vad.ts (agents-js/agents/src/inference/vad.ts at main · livekit/agents-js · GitHub)) on 8 kHz telephony that’s a lossy upsample before the model sees a sample. And 0.5 activation is high for band-limited audio; 300–3400 Hz systematically depresses Silero scores.
So don’t chase the raw probability, lower the gate:
new inference.VAD({ model: 'silero', activationThreshold: 0.3 })
deactivationThreshold follows automatically at activation - 0.15 unless you set it, and minSpeechDuration: 50 means you need two consecutive windows over the line at 0.4708 you were missing on both counts.
On your other two: the model is compiled into the native @livekit/local-inference binary and the version isn’t surfaced, but the hard-coded 512-sample window is the v5 contract (v4 also accepted 1024/1536 at 16 kHz). And no, don’t reset between turns resetState() already calls _nativeVad.reset() when a flush sentinel reaches the input stream, and the sliding pre-roll depends on continuity. Resetting per turn costs you the prefix padding that START_OF_SPEECH hands to STT.
Worth noting the 0.1386 figure is from replaying a whole call through one instance, which isn’t the live path I’d re-measure before treating it as what the agent actually saw.
Have you tried using different VAD providers? We have just released two new VAD models at ai-coustics and a custom LiveKit plugin that let’s you wire them up in your LiveKit app.
Hey @chase_fagen the repo mentioned above should have everything you need to setup the VAD 2.0, and you will also need ai-coustics SDK key. You can get it from the same link where the dev playground is
