I have been using Gemma 4 via LiveKit Inference ever since it launched, but recently, my agents started hitting ambiguous que times during calls resulting in 2000ms+ latency just for the LLM. Has anyone been dealing with this? Is LiveKit planning to address this or expand capacity to fix this?
What is the session ID for a session that had the issue? If you have agent insights enabled, can you share that session with LiveKit staff too? Agent logs would he helpful too.
The agent itself is self-hosted. But everything else is through LiveKit Inference.
Best sessions to look at (room names, all 2026-08-17 UTC):
- lk_SCL_S5N4wGJFwaKB, 14:43:46–14:45:24Z. Five consecutive LLM requests at 1620 / 2320 / 2790 / 2850 / 1640 ms TTFT.
- lk_SCL_K6N8yAF3r6c4, overlapping the same seconds, ≥0.99 cache hit: 16 requests, all 210–1060 ms. Same prompt at the same time with very different TTFT.
- lk_SCL_pCEm6mQDWRG3, 15:49–15:51:30Z. Six requests at 1.5–3.8 s, including two that came back with prompt_cached_tokens=0 and then re-healed to ≥0.997 cached on the next request in the same session.
TTFT p50 408 ms but p90 1040 / p99 2817. Decode is normal (~105 ms), the entire variance is pre-first-token, and slow requests come in 1–2 minute runs within one session.
Looking at your project, it looks like you have agent observability disabled, which makes it more difficult to investigate issues like this.
How are you measuring the TTFT? If you could share OTEL traces, that might also help.
The best solution would be to enable Agent Observability, reproduce the issue, and then share the observability data for those sessions. Although there is a small cost associated with Observability, all plans do come with a limited free allowance: Pricing | LiveKit
Okay, I enabled observability will provide that data tomorrow. But the rate is holding across multiple days, roughly 4.3% of call turns have that queueing signature and it causes them to have over 2000ms TTFT on Gemma 4.
i have raised similar issue here.
Its probably that gemma is used more often and is receiving more traffic and latency is increasing.
Agreed 100%. The 2000ms TTFT figure stood out to me though since that’s below even the P99 values on that benchmarks page.
I’ve seen similar results when using Gemma 4 31B through LiveKit inference. From the session logs the model seems to fall back to deepinfra as the inference provider, which results in higher latency as well as considerably worse output - there seems to be an issue with the checkpoint that deepinfra is serving, I’ve seen several issues with the model claiming a tool call that wasn’t called or even duplicating its response like here:
Likely offloading to deepinfra due to increased demand as already mentioned.
The LLM Inference class does have a parameter to allow specifying a provider, @darryncampbell do you know how we can set this to only get the LiveKit-hosted checkpoint and avoid the fallback? I’d much rather be able to control the fallback myself through a FallbackAdapter.
Love the model when it doesn’t fall back - and I was planning to move my production deployment to the LiveKit Cloud Ship plan to get reliable access to it - but without being able to prevent it falling back like this it’s essentially unusable.
@Paul_Barnes even if we fallback internally you should still consider that a LiveKit provided model - if our fallback deployment in this case was not performing as well as expected that is on us to fix and I’ll raise that with the team.
You can still use the FallbackAdapter in the agent but ideally you should not have to worry about it, since Inference will handle it for you.
Okay, thanks @darryncampbell - appreciate you raising it with the team.
Happy to test it again once a fix is in place, but FYI having an uncontrollable internal fallback mechanism like this would still render the model unusable for me. If I can’t trust a green eval suite when evaluating the model because production traffic might fall back to a deployment with a different configuration or behaviour, then I can’t really rely on those evals for production purposes.
A fallback opt-out, or a way to explicitly call the fallback deployments for evaluation purposes, would be necessary for me personally.