yeah i had set timeout to 2.5s because of high timeouts, but it should go to fallback only if within 2.5s it does not generate output? and as per gemma ttft in blogs and docs, it should never hit that 2.5s because claimed ttft is around ~200?
We also have a benchmarks page: LLM Latency Benchmarks | LiveKit
Thanks, I’ll let @Adrian_Cowham comment as he’s far more knowledgeable than myself but from what I can see, I would not have expected so many timeouts.
Hey Kaushal,
We’re working on some fixes that should be released on Monday. Here’s an update: we’re still seeing a some increased latency from gemma during peak usage. We solve this by routing requests to a backup deployment when requests start to slow down. We’ve identified an issue in the backup deployment, reasoning was enabled by default…causing increased TTFT.
Here’s my explanation, and the current thread we’re chasing for the increased timeouts you’ve been seeing.
- Your request comes to Inference
- Inference routes to Gemma-primary
- Gemma-primary takes too long for first token (no reasoning)
- Inference routes request to Gemma-backup
- Gemma-backup response quickly with first token (decides it needs to use reasoning)
- First token is a fast reasoning token
- Inference chooses Gemma-backup and cancels Gemma-primary
- Reasoning tokens are ignored on client and by the FallbackAdapter
- Gemma-backup takes a long time to reason and send the final response
- Timeout on client
Thanks! It would be great if the fix could be released on Monday i.e today .
I’m noticing fewer timeouts now. We had 99 timeouts on Saturday and only 4 on Sunday. I suspect this is because of the weekend, as historically we’ve seen lower LLM TTFT and fewer timeouts during weekends.
I’ll monitor it closely throughout this week and let you know whether the issue is resolved after the fix, or if we’re still seeing timeouts.
Thanks Kaushal, I’m working on getting the release out today. Really appreciate your patience and attention to detail on this.
Kaushal,
This has been rolled out. I will monitor our latency stats. Let me know how things look on your side.
Sure, I’ll continue monitoring and share the combined stats from Monday through Wednesday. I can already see some improvements. I just want to make sure this isn’t one of those occasional “good” periods we saw before, where timeouts temporarily dropped for a day or two before another spike.
Hi Adrian,
We’ve seen significant improvements. We had 25 LLM timeouts on 11 August, and only 6 timeouts so far today. While the issue isn’t fully resolved yet, the reduction is encouraging, and it definitely looks like things are moving in the right direction.
I will continue monitoring it throughout this week and would share some sessions with timeouts if helpful.
Also to add, we have LLM attempt_timeout set to 1.5 seconds (default is 5.0) in FallbackAdapter.
Hey Kaushal,
Thanks for the update! We’ll continue monitoring things on our side as well. Sharing some sessions with timeouts would be helpful, looking at the traces would be helpful.
Thanks!
The timeout have dropped significantly, however not zero
Sample session with issue = https://cloud.livekit.io/projects/p_3scf9tt1e2i/sessions/RM_k9zt7eZ585gh/observability?mode=transcript
https://cloud.livekit.io/projects/p_3scf9tt1e2i/sessions/RM_ENcQSipVGxda/observability?mode=logs&t=1786716933639 (without traces) but log has all timeout details
11 aug = 25
12 aug = 71
13 aug = 5
14 aug = 16 (till now)
Thanks Kaushal, I will dig into this. Thanks for the sessions.
Update as of now:
| Date | Count |
|---|---|
| 11 Aug | 25 |
| 12 Aug | 71 |
| 13 Aug | 5 |
| 14 Aug | 30 |
| 15 Aug | 2 |
| 16 Aug | 0 |
| 17 Aug | 0 (till now) |
The issue appears to be resolved i think. I will check throughout this week.
However, performance isn’t quite as fast as documented ( Gemma 4 31B on LiveKit Inference | A faster, cheaper default for voice | LiveKit ).
To be fair, it’s already quite good — I’m seeing an average TTFT of around 250–510ms. That said, if it could match the numbers mentioned in the documentation, it would be excellent.
Thanks for all the fixes and updates till now
The size of your system prompt can affect response times. Also they way you structure your system prompt can matter. Keep “dynamic” data at the end of the system promp when possible.
Yes i agree!, so its already like the way you mentioned, dynamic data at end to have caching better, we have around 75-80% of cached tokens, also our prompt size is around 2k on an avg i think which is normal for gemma model. So considering all this, i was expecting the documented speed.
Curious which “documented speed” you mean? Are you not seeing it outperform GPT-4.1 mini, Gemini 2.5 Flash, … on the same prompt?
Sorry what i meant was, as per document here Gemma 4 31B on LiveKit Inference | A faster, cheaper default for voice | LiveKit
it should ideally be ~200ms ttft, so very few sessions have < 300, but most of them have higher than 350ms in ttft. So yes its faster and outperform GPT-4.1 mini, Gemini 2.5 Flash, … on the same prompt but its not the same latency that is mentioned.
was curious about what should we expect in terms of latency.
It matches as in benchmarks here-LLM Latency Benchmarks | LiveKit
So ideally the above doc must be updated? to showcase it around with real benchmarked latency?
I am not sure what “prompt” they used for that benchmark. But I think the important part of that doc is the relationship between the different models and not the exact number. Different prompts, tools, etc will have different effects on the models response times.
Thanks, I agree, not an issue. I was just curious about the exact number we should expect. It’s mainly so I can understand the baseline. It also helps us debug more effectively if we start seeing higher latency or unexpected performance, as we’ll have a better idea of what the expected numbers should look like and whether there’s something we need to investigate on our end.
This is the best answer I have on the topic. It continues to run so you can see history of the benchmark over time.