STT via LiveKit Inference (Deepgram): two issues on SIP/phone calls

Getting Started
1

Two STT issues with inference.STT (Deepgram) on SIP/phone calls

Setup

  • livekit-agents — cascaded mode, SIP/phone outbound call
  • STT: Deepgram Nova-3 via inference.STT, language="hi"
  • TTS: Sarvam Bulbul v3 (direct plugin, not inference)
  • Turn detection: MultilingualModel, max_delay=1.0s, adaptive interruption enabled
  • NC: krisp_enabled=True on the SIP participant (no in-agent NC object)
  • aec_warmup_duration=1.5, min_consecutive_speech_delay=0.30

Issue 1 — ~4s STT dead zone after agent’s first TTS playback

After the agent finishes its opening TTS utterance and enters listening state,
Deepgram stops emitting any events — including heartbeat/metrics — for ~3.9 seconds.
Any user speech during this window is completely dropped with no interim and no final
transcript produced.

What we see in the traces

34.71s Agent enters listening (first TTS done)
34.87s STT metrics ← heartbeats firing normally
35.94s STT metrics
37.84s STT metrics
38.69s STT metrics ← last event before gap
[user spoke here — zero Deepgram output for 3.9s]
42.59s STT metrics + [STT FINAL] ‘Hello.’ ← user’s repeat captured

The STT heartbeats were firing every ~1–2s before the gap and resumed normally
after it. The gap only appears after the first TTS playback of the session —
subsequent TTS → listening transitions do not show the same dead window.

Question

Is there a known reconnection or buffer-flush behaviour in inference.STT that
causes this dead window after the agent’s first TTS output on a SIP call?
Is there any way to detect this state or warm the STT connection before the agent
speaks its opening line?

Issue 2 — Deepgram interim never finalizes; stream only flushes at disconnect

When the user speaks during the agent’s listening state on a phone/SIP call,
Deepgram produces an interim transcript but never emits a final — keeping the
turn open for 15+ seconds. The final only arrives when the STT stream is torn down
at participant disconnect.

What we see in the traces

53.99s Agent enters listening
56.45s STT metrics
60.54s STT metrics ← user spoke around here, no transcript
65.54s STT metrics
67.54s [STT interim] ‘thoda’ ← Deepgram heard the user
68.04s STT metrics
69.00s Inactivity timer fires (15s expired — no STT FINAL arrived)
[agent speaks inactivity warning]
72.76s Agent re-enters listening
73–77s STT metrics ← Deepgram still processing
82.83s Participant disconnects
82.91s [STT FINAL] ‘thoda’ ← stream flushed at disconnect, ~15s after interim

Hypothesis

The inactivity warning’s TTS audio echoes back through the PSTN path. Since NC is
handled at the SIP trunk level and not inside the agent, Deepgram’s endpointing
sees continuous audio and refuses to close the utterance. It keeps accumulating
audio from both the user and the echoed TTS, never reaching a silence threshold to
emit the final.

This creates a compounding problem: the inactivity timer fires because no final
arrives → agent speaks → that TTS echo prevents the next final → repeat.

Questions

  1. On SIP/phone calls, is there a known issue with inference.STT (Deepgram)
    keeping transcripts in interim state indefinitely when TTS audio echoes back
    through the PSTN path?

  2. Is there an utterance_end_ms or equivalent endpointing parameter exposed
    through inference.STT to force finalization after N ms of continuous speech?

  3. Is there a recommended way to handle AEC on the agent side when krisp_enabled
    is set at the SIP trunk level — specifically to prevent TTS echo from confusing
    Deepgram’s endpointing?

What we have tried

  • krisp_enabled=True on the SIP participant to handle NC at the trunk level
  • aec_warmup_duration=1.5 to suppress early echo
  • language="hi" (not "multi" — multi hallucinated non-Hindi languages on
    Hindi speakers)

Any pointers appreciated.

2

What version of LiveKit agents are you using? I would also recommend using the latest turn detector model, but I’m not saying that is the root cause of the issues you are describing.
Issue 1: There aren’t any known issues for a dead window after the first TTS. Is the turn being detected? That should show up in the agent observability trace.

Issue 2: Any reason you are applying noise cancellation on the trunk and not the agent? We recommend applying nc at the agent level for agent-telephony use cases. I would definitely set the nc on the agent first but another thing to consider is whether you are setting any endpointing value on the STT, since that will affect your final transcript output.

3
  • Version: livekit-agents 1.5.8, livekit-plugins-turn-detector 1.5.8.
  • Issue 1 follow-up: Yes, EOU/turn detection events are present in the observability trace — they appear normally before and after the dead window. The gap is specifically at the Deepgram/inference.STT level: no interim, no final, no STT metrics at all for ~4s. Since the turn detector (MultilingualModel) only processes what STT produces, it has nothing to work with during this window. The dead window appears consistently only after the agent’s first TTS output of the session, not on subsequent turns — which makes us suspect a reconnection or buffer-reset on the inference.STT connection after the initial audio flush.
  • Issue 2 follow-up — NC: The reason we apply krisp at the SIP participant level (krisp_enabled=True) rather than agent-level BVC() is to avoid double noise cancellation — ai_coustics and BVC are both trained on raw audio and produce artifacts when layered. For web calls we use BVC() in the agent, but for phone we disable in-agent NC entirely. If you recommend agent-level NC for telephony, should we disable NC at the trunk and use BVC() in-agent instead? Or is there a way to layer them safely?

On endpointing — we’re currently passing no endpointing parameters to inference.STT. We haven’t set utterance_end_ms or equivalent. Our concern is that even with a forced endpointing timeout, the TTS audio echoing back through the PSTN path would keep resetting Deepgram’s silence clock, so the interim would still never finalize. Is there a way to gate endpointing on speech energy rather than silence duration for phone calls?

4

Thanks, that all makes sense. For the question on noise cancellation, I wrote this recently but essentially the recommendation is to remove the trunk noise cancellation and only have it in the agent:

If you can share the call IDs (SCL_) or session IDs I will see if there’s anything in our server or inference latency logs that might explain the issues

5

Sure sharing the session id in DM

6

It is late for Darryn now. You can send it to me if you like.

Also if you can share the agent insights, and if you recorded egress it would also be helpful to see that too.