I’m building a voice marketing agent using Gemini 3.1 Flash Live (Realtime API) as the voice model, connected via SIP through LiveKit. Since I’m using the Realtime/Live model, STT, conversational reasoning, and TTS are all handled inside the same Gemini session — I’m not using separate STT/VAD/TTS components.
Issue: When the agent is mid-sentence and the user speaks (even briefly, e.g. a stray “hello”), the agent’s audio cuts off abruptly at that exact word. If the user then speaks again shortly after the agent resumes, the same cutoff happens again — producing a repeated stutter/break pattern through the conversation rather than a clean interruption.
I captured the raw PCM audio Gemini sent during one of these interruptions and compared it against the actual call audio. Gemini had already sent more audio past the point where the user interrupted (the sentence continued well beyond the word where the live call cut off) — but that audio was never played to the caller. Based on this, it looks like LiveKit’s playout pipeline is discarding already-received-but-not-yet-played audio frames on interrupt (buffer clear on SpeechHandle cancel), rather than letting the in-flight response finish playing out before stopping.
Question: Is there a way to configure LiveKit (or the Gemini Realtime integration specifically) so that once a response has already been generated and is in the outbound buffer, it plays out to completion rather than being discarded on interrupt? I understand this may be intentional default behavior for snappy interruptions in text/chat-style use cases, but for a live phone call it produces jarring mid-word cutoffs. Is there a “graceful drain instead of hard clear” option, or a recommended pattern for handling this in a telephony context?
The agent stops speaking because they were interrupted
Any audio in the pipeline that the agent had not already spoken is discarded
You would like to modify this behaviour so that the agent audio that has not already been heard is retained.
This behaviour is by design, since the assumption is you only want to retain the audio the user actually heard. Otherwise your agent context will not match the user’s understanding of the conversation.
Is the actual issue here that the agent is being interrupted too easily? And the agent stops speaking unnecessarily as part of the normal conversation? If so, you might try enabling LiveKit’s turn detector which works with Gemini, as described here:
Thanks, Darryn — that’s really helpful and confirms what we saw in the PCM capture.
Our issue is a mix of genuine user barge-ins and the agent being interrupted a bit too aggressively on short filler words. We don’t want to disable interruption handling, just make recovery after a real interrupt feel smoother.
Before trying TurnDetector with automatic_activity_detection.disabled=True, I had two questions:
Language support: Our agent is Kannada/Kanglish, and kn doesn’t appear to be supported by TurnDetector. Does it fall back to a generic model, or would detection quality likely degrade compared to Gemini’s server-side activity detection?
generate_reply() interaction: I noticed livekit/agents#5408 mentioning issues with AAD disabled and generate_reply(). Is that still relevant, especially for setups where we occasionally inject prompts programmatically?
Given this, we’re leaning toward improving recovery after real interruptions first, then evaluating the TurnDetector approach in a separate branch once the current flow is stable.
Interesting. Given what you’ve said, I agree with your approach. It’s strange the documentation does not list any configuration for the built-in turn detector, I did some tests with the automatic_activity_detection, https://ai.google.dev/gemini-api/docs/live-api/capabilities but I don’t see much difference in behaviour - it looks like the framework is passing the values through:
llm=google.realtime.RealtimeModel(
model="gemini-3.1-flash-live-preview",
voice="Puck",
realtime_input_config=types.RealtimeInputConfig(
automatic_activity_detection=types.AutomaticActivityDetection(
start_of_speech_sensitivity=types.StartSensitivity.START_SENSITIVITY_HIGH,
end_of_speech_sensitivity=types.EndSensitivity.END_SENSITIVITY_LOW,
prefix_padding_ms=200, # audio kept before speech start is committed
silence_duration_ms=800, # trailing silence before the turn is closed
),
),
),
To answer your questions:
Language support: Our agent is Kannada/Kanglish, and kn doesn’t appear to be supported by TurnDetector. Does it fall back to a generic model, or would detection quality likely degrade compared to Gemini’s server-side activity detection?
No, it would not fall back to a generic (VAD-based) model, I think it would just ‘do its best’ with the audio it has. Just because the turn detection model was trained on specific languages, it doesn’t mean it wouldn’t work well with similar languages. So, I think it’s worth a try, but I’m not sure how well it would work.
generate_reply() interaction: I noticed livekit/agents#5408 mentioning issues with AAD disabled and generate_reply(). Is that still relevant, especially for setups where we occasionally inject prompts programmatically?
Thanks for testing that and for the detailed answer — very helpful on both counts.
The Gemini 3.1 compatibility note is honestly the more important finding for us right now. We’ve built our entire flow-control system today around injecting mid-session instructions via send_client_content (telling the model to speak specific locked lines at specific points — greetings, recovery re-asks, etc.), precisely because generate_reply()/update_instructions() didn’t work for us either. I hadn’t seen this documented as an official 3.1 limitation until your message — I assumed it was just this specific plugin version’s quirk, not a documented model-level restriction with a 1007 rejection after the first turn.
I’m going to go verify directly whether our injects are actually succeeding or being silently rejected — if some of what we thought was working today was coincidental (the model doing the right thing on its own) rather than caused by our injects actually landing, that would explain some inconsistent behavior we’ve seen. Is there a public tracking issue for the “long-term fix being investigated” mentioned in the docs, so we can watch for updates? And separately — do you know if there’s any recommended workaround pattern for getting deterministic mid-call instructions into a 3.1 session today, given send_client_content is restricted to initial history seeding only?
On the TurnDetector/Kannada point — that’s a fair characterization, “do its best, unknown quality” is honestly what I expected. Given the 3.1 compatibility issue you just confirmed is unfortunately still open, I think we’ll hold off on the TurnDetector experiment for now and focus on making sure our current setup’s mid-call instruction delivery is actually reliable — will follow up if we find something concrete on that front.
Is there a public tracking issue for the “long-term fix being investigated” mentioned in the docs
Yes, it dates from when we first launched Gemini 3.1 and is here:
The best conversation in the community at the time is here:
Obviously you are aware of this possibility, but another option is to explore a different model, or (pipeline) architecture. This would give you a lot more flexibility and choice.