Gemini Realtime latency spikes and unexpected call termination

Telephony
, , ,
1

We are seeing frequent latency spikes when using Gemini Realtime with LiveKit Agents in a SIP outbound calling setup (Both Plivo and Twilio).

User speech detection appears normal, but model TTFT frequently spikes to 10 -15 seconds, causing long pauses before AI responses.

We are also occasionally seeing abrupt call disconnections while the conversation is still active, without an explicit end-call trigger.

Any guidance on possible causes (e.g., session context growth, streaming delays, or known issues with Gemini Realtime) would be appreciated.

2

Hi, I see you also reposted to here, High latency (5-8 seconds) with Google Gemini Realtime plugin over SIP - #2 by LiveKit-Community. Whilst that ticket is tagged for self-hosting LiveKit (and I see you are using LiveKit cloud), the advice there (to compare your performance with an alternative realtime model) still holds. At the very least, it would help you isolate the root cause of the latency.

Interested in others’ thoughts - there are folks in this community with a lot more experience with Gemini Realtime than me!

3

Hello Krishna,

I totally get how frustrating those 10–15 second delays and unexpected call drops can be in your Gemini + LiveKit + SIP (Plivo/Twilio) setup. From what I’ve seen, this usually boils down to a few key issues:

Possible Causes:

  • Gemini’s backend might be experiencing cold starts or hitting rate limits, causing delays.

  • LiveKit Agents could be bogged down by heavy session loads or inefficient processing.

  • Plivo/Twilio may be terminating calls if there’s no audio or response detected within ~30 seconds.

Here’s How You Can Debug It:

  1. Isolate Gemini:
    Try testing Gemini independently (without LiveKit or SIP) to see if the delays are coming from Gemini itself.

  2. Check LiveKit Logs:
    Look for any errors or slowdowns in LiveKit Agents during AI response generation.

  3. Enable SIP Debugging:
    Use Plivo or Twilio’s SIP debugging tools to identify why calls are dropping—could be timeouts or signaling issues.

  4. Keep the Call Active:
    Send silent audio packets during AI processing to prevent the SIP provider from timing out the call.

Next Steps:
If you’re still running into issues, feel free to share logs from LiveKit, Gemini, and Plivo/Twilio, and I’ll help you dig deeper. Also, double-check that you’re using the latest version of LiveKit Agents to avoid any known bugs.

Honestly, if you’re dealing with these issues often, Plivo could be a game-changer. Our low-latency SIP trunking and AI-friendly call handling are built to keep things running smoothly so you won’t have to worry about dropped calls or laggy responses. If you’re curious, I’d be happy to help you switch things over and get everything working perfectly. Let me know..