High response latency after short user utterances despite preemptive generation and aggressive endpointing (Agents 1.6.4)

Getting Started
1

Hi LiveKit team,

We’re using LiveKit Agents 1.6.4 with a self-hosted LiveKit server for a telephony voice agent.

We’re seeing a consistent 2–3 second delay after the user responds with short utterances such as:

  • Hello
  • Yes
  • Okay

This creates a poor phone experience because users often start saying “Hello… Hello…” while waiting for the agent to respond.

Configuration

turn_handling = TurnHandlingOptions(
    turn_detection=inference.TurnDetector(),
    endpointing={
        "min_delay": 0.3,
        "max_delay": 1.0,
    },
    preemptive_generation={
        "enabled": True,
        "preemptive_tts": True,
    },
)

AgentSession(
    turn_handling=turn_handling,
    vad=vad,
    stt=deepgram_stt,
    llm=llm,
    tts=tts,
)

Stack

  • LiveKit Agents: 1.6.4
  • LiveKit Server: Self-hosted
  • STT: Deepgram Nova-3
  • LLM: google/gemma-4-31b-it via LiveKit inference /. also test with gpt-4.1-mini
  • TTS: ElevenLabs

Logs

For a simple user reply:

User stopped speaking
↓

trans_delay = 0.26s
eou_delay = 1.00s

↓

LLM started

↓

LLM completed in 2.22s

↓

TTS first audio = 0.21s

Overall response latency is around 3.4 seconds.

Another example:

trans_delay = 0.55s
eou_delay = 1.00s
LLM = 0.73s
TTS = 0.46s

≈2.2 seconds total

Observations

  • STT latency looks good (250–500 ms).
  • TTS latency is also good (200–450 ms).
  • The delay seems to come from:
    1. eou_delay always reaching 1.00s
    2. LLM inference time.

What concerns us is that eou_delay is always exactly 1.00s, even for one-word responses like “Yes” or “Okay”, despite:

min_delay=0.3
max_delay=1.0

We expected many of these utterances to commit closer to min_delay.

Questions

  1. Is it expected that TurnDetector() waits until max_delay for short acknowledgements?
  2. Are there additional settings that allow earlier commitment for short responses?
  3. Is preemptive generation expected to begin before the final end-of-turn? In our logs, LLM generation appears to start only after the final transcript is committed.
  4. Would you recommend using a different endpointing mode (e.g., dynamic endpointing) for telephony?
  5. Is there a recommended configuration for achieving sub-second response latency in voice calls?

Any guidance would be greatly appreciated.

Thanks & Regards,
Anil Mohite

2

Can anyone help with this short user utterance issue? It’s the major issue we are currently facing.
In Deepgram Nova 3, I kept endpointing at 50ms, but it still doesn’t handle EOU properly. I want to reduce that time. Also, the LLM is taking time to revert.

3

Hi, detecting the end of turn for these short utterances is a notoriously difficult problem to solve. For use cases there you ONLY expect the user to reply with single words, it’s often best to just use VAD for turn detection, but for natural speech it’s more difficult to detect the odd short word correctly.

Just to clarify, you say you are using self-hosted LiveKit server, but I believe you are using LiveKit Cloud, with self-hosted agents. That gives you access to the v1-mini turn detection model, which I would recommend, and from your configuration, it seems as though you are already using it.

Is it expected that TurnDetector() waits until max_delay for short acknowledgements?

With fixed endpointing, yes since the model likely couldn’t determine from the single word that the turn had ended. You could try dynamic endpointing if you had frequent short words, Turn handling options | LiveKit Documentation, that would lower the max delay based on the session to date.

Are there additional settings that allow earlier commitment for short responses?

Nothing specific that only applies to short responses. If your workflow had a period where you expected the user to reply with short answers, you could update the endpointing at runtime: Turn handling options | LiveKit Documentation

Is preemptive generation expected to begin before the final end-of-turn? In our logs, LLM generation appears to start only after the final transcript is committed.

The best way to see the effect of preemptive generation (in my opinion) is to use Agent Insights. I attached a screenshot to this issue the other day that shows where to look: LiveKit preemptive generation not triggering with current STT model / missing preflight_transcript events. - #2 by darryncampbell

Is there a recommended configuration for achieving sub-second response latency in voice calls?

If you mean overall response time, the best resource I can point you to is this:

If you mean sub-second for turn detection, it looks from your configuration that you are doing everything correctly in your config (besides what I mentioned above), you’re just hitting issues with short utterences.

4

Thanks @darryncampbell , Dynamic endpointing is works better for short utterances.

endpointing={

  "mode": "dynamic",

  "min_delay": 0.25,

  "max_delay": 0.8,

  "alpha": 0.9,

},