LiveKit Inference timeouts happening since past 2 weeks

[Kaushal_Shah]

This is how gemma model is used

  return inference.LLM(
    model=f"google/{cfg.model or _DEFAULTS['google']}",
    extra_kwargs=extra, # with temperature 0.7 and max_completion_tokens 250
  )

Tiemouts are captured live from the Agents SDK event callbacks in our worker:

• Errors — error events on each adapter plus the AgentSession “error” event (per your “Events and error handling” doc pattern), recorded as {type, message, recoverable, phase}.
• Fallbacks — stt/llm/tts_availability_changed events; we record every transition and whether the fallback ever engaged.

On timeouts specifically — we don’t have a single “timeout” metric; they show up three ways:

1. LLM attempt timeout — our LLM FallbackAdapter runs with attempt_timeout=2.5s (I was facing same issue with default 5.0. so i reduced as it was causing more latency). A slow primary triggers llm_availability_changed and a switch to fallback, so these appear as fallback engagements.
2. Provider API timeouts — surface as error events with type APITimeoutError (or provider equivalent); we count them from the recorded error type.

So we have been using livekit from past 2 months, and issue started from 14th july, for 5.4 mini openai model and from last week we have switched to gemma model as it was more faster.

Note that on some days timeouts dropped significantly and almost 0-4, while on other days its much higher, and traffic/sessions are mostly uniform on all these days of july