We run production voice agents on livekit-agents 1.6.x (self-hosted server, standard STT→LLM→TTS pipeline) and we’ve repeatedly hit a failure mode across multiple STT plugins — Deepgram (Nova and Flux),
AssemblyAI, and most recently Sarvam:
The STT stream connects and transcribes fine for the first few turns.
Then it silently stops emitting transcripts — no close frame, no error, no exception from the plugin. The websocket looks half-open.
Audio keeps flowing in, local VAD confirms the user is genuinely speaking, but no interim/final transcripts arrive.
The caller keeps talking to an agent that has gone deaf. From the user’s side: “the bot answered 2–3 times, then stopped working.”
Because AgentSession gets no error event, nothing in the framework reacts — the session just sits there until the user gives up or an inactivity timeout fires.
We’ve built our own recovery around it: a watchdog that treats “VAD-detected speech ended but no final transcript within N seconds” as a stall, then escalates — force-commit the turn, tear down and reconnect
the STT stream, optionally fail over to a backup STT provider, and as a last resort speak a goodbye and end the session cleanly. It works (took one deployment’s abandoned-call rate from ~23% back to single
digits), but it’s a lot of machinery, and the fact that we’ve needed it for three unrelated vendors suggests it’s a framework-level gap rather than a provider bug.
Questions for the community / LiveKit team:
Is anyone else seeing silent STT stream stalls on long-lived agent sessions? Which providers?
Is there existing SDK machinery we’ve missed that detects “audio in, no transcripts out”?
Would the team consider a built-in STT liveness watchdog in AgentSession (or the STT base class) — e.g. a configurable stall threshold that emits an error/triggers the existing FallbackAdapter — so every
operator doesn’t have to build this independently?
Any known-good patterns for proactive websocket rotation on long sessions?
Happy to share logs/timings from affected sessions.
Hi Troy, welcome to the community. It’s good to find you again - I was trying to link someone to your dashboard the other month who was asking about the feasibility of deploying LiveKit in Australia, but I could only half-remember where I had seen it.
Anyway, I believe this PR covers what you are asking about? I didn’t realise it was so close, but it looks like it just got merged a few hours ago, so I expect it to be in the next release.
Hi David, thanks for the reply and the pointer, and I’m glad the livekit dashboard, stuck in your memory! Yes, using the dashboard, we are also running SAAS LiveKit here in Australia, and it’s been great for exactly that use case.
That PR is very timely, and it does cover the detection half of what I was asking about — a first-class signal that VAD heard the caller but STT produced nothing.
Worth noting for anyone else following along, though: it’s detection only. When the timeout fires, the SDK emits an event and takes no action itself, reading the PR discussion, the team deliberately kept it opt-in and event-only because of false-positive risk from background noise, so recovery stays the application’s job.
We ended up building that recovery layer ourselves after seeing silent WebSocket stalls across multiple STT providers (VAD-proven caller speech, zero transcripts, no error and no close frame — so the SDK’s
retry ladder never fires).
Our approach: a stream-liveness watchdog, a thin STT wrapper that forces the dead stream to raise so the SDK’s own reconnect ladder rebuilds it, mid-call failover to a backup STT provider, and a graceful goodbye + hangup if everything’s dead.
It’s good to see the SDK now offering a native signal to hang that on; once this lands in a release, we’ll likely wire the new event in as an additional trigger alongside our watchdog or the reply.