Turn-taking on real SIP calls: what cut my median reply from 2.44 s to 1.81 s (plus a question on short answers)

Agents
, , , , , , , , ,
1

I’ve been measuring a LiveKit Agents phone agent (Python, SIP, Deepgram, Gemini, Cartesia) turn by turn, and wanted to share a few findings in case they help others, plus one open question.

What moved the numbers (agent-side, 63 replies over 10 calls):

  1. Turning off Gemini 2.5 Flash’s thinking via LiveKit Inference: extra_kwargs={“reasoning_effort”: “none”}. First token went from ~1.1 s to ~0.5–0.9 s from India. Biggest single win.

  2. Endpointing: max_delay 1.3 s was fast but cut callers off while they paused to think (7 cut-offs in one call). {“mode”: “dynamic”, “max_delay”: 2.0} gave zero cut-offs.

  3. A small wrapper around the turn detector stream: if the audio model is unsure and the transcript is a bare “yes”/“no” right after a question, it raises the end-of-turn probability. Gotcha: the audio prediction usually lands ~0.1 s before Deepgram’s final transcript, so the wrapper waits up to 0.6 s for it, only when the model is unsure. Since the endpointing delay counts from end of speech, that wait costs nothing.

  4. Fallback across LLM providers: a reply from one provider carried message metadata (extra) that the other provider rejected with a 400 (“extra_content is unsupported”), which broke every later turn. Stripping extra from the history before each request fixed it.

  5. A very fast model (0.16 s first token) intermittently returned empty or mid-sentence replies with no error. Empty ones are catchable before TTS; half-sentences aren’t, since they stream straight to speech.

Also worth knowing: on the phone itself, the same calls heard ~2.8 s median vs 1.5 s at the agent. The free softphone SIP path adds ~1–1.5 s the agent never sees.

Code: GitHub - mohammed-usmani/voiceAgent: Phone voice agent (LiveKit, Python) that runs job pre-screening calls, logs every turn-taking decision, and measures reply latency per turn. · GitHub
Demo (a real call, with the agent-side breakdown at the end): #voiceai #conversationalai #aiagents | Mohammed Usmani

Question: how are others handling hesitant one-word answers on phone audio? A flat “No.” still often scores below the English threshold (~0.56) on the audio turn detector. Is a text-side rule like mine the usual approach, or is there a better signal I’m missing?

2

Thanks for sharing with the community.

I hope to be able to share something about this soon.

3

Thanks Darryn, great to hear! Happy to help test it when it’s ready. My decision logs have a few dozen real short-answer cases from phone calls (turn detector scores, what the caller did next, and whether the agent waited or cut in), in case examples like these are useful. The longer write-up with the numbers is here: