Turn detection for non-native speakers: long mid-sentence pauses, not short utterances

Agents
,
1

I am currently building a voice agent that simulates the B1 German oral exam, for people like me preparing for the test but could not really find a good tool. The users are language learners, so almost none of them are native speakers.

The post on debugging short utterances was useful, but my problem turned out to be on the opposite side.

A B1 learner does not produce utterances that are too short. They stop in the middle of a sentence for two or three seconds while they search for a word(you know that), and then carry on. With the default settings the agent treated that pause as the end of the turn and answered half a question. For this product that is worse than a slow agent, because being interrupted while you are already nervous is exactly the thing it is supposed to fix.

What I ended up with:

  • min_delay 1.0 s instead of 0.5 s
  • max_delay 5.0 s instead of 3.0 s
  • LiveKit’s turn detector deciding in between

In my own sessions it holds the turn for the full 5 seconds on roughly one turn in five. So the model is genuinely uncertain a lot of the time with this user group.

There are two things I have not solved:

1. A 5 second worst case is a long wait when the user really has finished. Is anyone adapting endpointing per user rather than per session? Someone who hesitates a lot and a confident speaker need different settings, and I can tell them apart after a few turns.

2. Learners almost never backchannel. No “mhm”, no “genau”. I assume backchannels are one of the last things you acquire in a new language. Has anyone else seen this, and would it change how you configure interruption handling?

Definitely happy to share more detail if it is useful!

2

@neimuc27 On the per-user question: 1.8.11 has this built in, and it’s easy to miss. Set endpointing to dynamic:

AgentSession(
    turn_handling={
        "endpointing": {"mode": "dynamic", "min_delay": 0.5, "max_delay": 5.0},
    },
)

It learns the user’s pauses during the session: the gaps between their utterances, and the cases where the agent answered too early and the user immediately kept talking. It raises min_delay towards their pause length, capped at max_delay. Two things to know from the code: it only learns upwards from the min_delay you give it, so keep that low and confident speakers don’t pay for the hesitant ones. And it starts again at min_delay on every new session. Since you can tell your users apart, store what you learn per user and start returning learners higher (session.update_options(endpointing_opts=...) changes it mid-session too). alpha (default 0.9) is how slowly it moves. Lower it if three or four turns is too long to adapt.

On backchannels, I think your observation matters more than it looks. LiveKit’s adaptive interruption handling spends a lot of its effort telling backchannels apart from real interruptions. With learners, almost all overlapping speech is real, so the risk flips. Check false_interruption_timeout: it defaults to 2.0 s. A learner who interrupts, then pauses two or three seconds to find the word, looks like a false interruption, and with resume_false_interruption on (the default) the agent carries on talking over them. Raising that timeout to roughly the length of your users’ pauses is worth testing.

One measurement note if you tune this further: score the pauses against where the speaker actually stopped, not the VAD’s own end-of-speech, or the numbers will look better than the experience is. Happy to share how I did that if it’s useful.

3

Interesting, the adaptive model would have been trained on native speakers. I wonder if you would get a better experience setting the interruption mode back to vad: Turns overview | LiveKit Documentation . My guess would be probably not, but it’s worth a test to see if it gives a better user experience for your use case.