I have been using Gemma 4 via LiveKit Inference ever since it launched, but recently, my agents started hitting ambiguous que times during calls resulting in 2000ms+ latency just for the LLM. Has anyone been dealing with this? Is LiveKit planning to address this or expand capacity to fix this?
What is the session ID for a session that had the issue? If you have agent insights enabled, can you share that session with LiveKit staff too? Agent logs would he helpful too.
The agent itself is self-hosted. But everything else is through LiveKit Inference.
Best sessions to look at (room names, all 2026-08-17 UTC):
- lk_SCL_S5N4wGJFwaKB, 14:43:46–14:45:24Z. Five consecutive LLM requests at 1620 / 2320 / 2790 / 2850 / 1640 ms TTFT.
- lk_SCL_K6N8yAF3r6c4, overlapping the same seconds, ≥0.99 cache hit: 16 requests, all 210–1060 ms. Same prompt at the same time with very different TTFT.
- lk_SCL_pCEm6mQDWRG3, 15:49–15:51:30Z. Six requests at 1.5–3.8 s, including two that came back with prompt_cached_tokens=0 and then re-healed to ≥0.997 cached on the next request in the same session.
TTFT p50 408 ms but p90 1040 / p99 2817. Decode is normal (~105 ms), the entire variance is pre-first-token, and slow requests come in 1–2 minute runs within one session.
Looking at your project, it looks like you have agent observability disabled, which makes it more difficult to investigate issues like this.
How are you measuring the TTFT? If you could share OTEL traces, that might also help.
The best solution would be to enable Agent Observability, reproduce the issue, and then share the observability data for those sessions. Although there is a small cost associated with Observability, all plans do come with a limited free allowance: Pricing | LiveKit
Okay, I enabled observability will provide that data tomorrow. But the rate is holding across multiple days, roughly 4.3% of call turns have that queueing signature and it causes them to have over 2000ms TTFT on Gemma 4.
i have raised similar issue here.
Its probably that gemma is used more often and is receiving more traffic and latency is increasing.
Agreed 100%. The 2000ms TTFT figure stood out to me though since that’s below even the P99 values on that benchmarks page.
I’ve seen similar results when using Gemma 4 31B through LiveKit inference. From the session logs the model seems to fall back to deepinfra as the inference provider, which results in higher latency as well as considerably worse output - there seems to be an issue with the checkpoint that deepinfra is serving, I’ve seen several issues with the model claiming a tool call that wasn’t called or even duplicating its response like here:
Likely offloading to deepinfra due to increased demand as already mentioned.
The LLM Inference class does have a parameter to allow specifying a provider, @darryncampbell do you know how we can set this to only get the LiveKit-hosted checkpoint and avoid the fallback? I’d much rather be able to control the fallback myself through a FallbackAdapter.
Love the model when it doesn’t fall back - and I was planning to move my production deployment to the LiveKit Cloud Ship plan to get reliable access to it - but without being able to prevent it falling back like this it’s essentially unusable.