We noticed a significant increase in GPT-5.4-mini timeouts when using LiveKit Inference since last two weeks
We’ve been running GPT-5.4-mini in production for our real-time voice agents, and it has been very stable until recently. Starting from 14th July, we began seeing a noticeable increase in timeout errors, with 21 July being the worst day, where we recorded around 70 timeout incidents in a single day.
The issue appears to be intermittent, most requests succeed, but some requests stall and eventually time out, impacting the voice agent experience.
We’re trying to determine whether:
Others are seeing similar behaviour.
This is specific to GPT-5.4-mini or affects other OpenAI models as well.
It would be helpful to know whether this is an isolated issue or something affecting multiple deployments.
We dont have observability enabled, but could share whole session report json if needed.
The issue is still happening, so on weekend saturday, sunday, it seems to have been fixed with about 7 timeouts, but today it again rose to 50+ timeouts in single day for gemma4 model livekit inference, its extremely dissapoitning as for same modes if i dont use livekit inference and directly use providers like openai, than issue seems to be resolved, so its issue in livekit inference. If issue does not resolves, i am thinking to switch stack away to different provider.
I am looking at the P95 TTFT latency over the past 14 days and the hosted model is consistently faster than gpt-5.4-mini, it’s not even close.
Looking at the 5.4-mini graph specifically, it’s difficult to say if any increase is statistically significant to be honest but I would not expect you to see any better figures by going to the provider directly - unfortunately, these things are very difficult to compare apples to apples.
Right now, I’m pretty sure you can combine Inference with plugins in the Fallback array, which might also help if you were considering a direct approach:
llm=llm.FallbackAdapter([
inference.LLM("openai/gpt-5.4-mini"), # LiveKit gateway (keyless, metered)
openai.LLM(model="gpt-5.4-mini"), # direct OpenAI, your own key
]),
Related, I know the team are making efforts to improve the fallback capabilities of our Inference models in general, which would help in this scenario, but I don’t see anything publicly announced yet I’m afraid.
Yes i am using livekit inference in cloud, so its hosted by livekit (gemma4) model, yeah i agree its faster than 5.4 mini (hosted by livekit) and have been using from last week, but the number of timeouts are huge, its only happening with livekit inference hosted models and not via third party. So its happening for all models hosted via livekit.
Attaching image, so its very serious issue, as livekit inference hosted models are only once that has this issue, and people who track timeouts can notice, otherwise fallback models will be used and it works fine.
Thanks, I see this thread got flagged internally so @Neil_Dwyer may have additional feedback.
I should have picked up on this with my earlier answer, but since we only serve the gemma model from from US, I’m sure that will be a factor in the latency (and explain why your dashboard doesn’t match the numbers I’m seeing internally). Please see the thread below that clarifies our plans to bring the model to the EU soon:
Hmm right, so actually its not happening just for gemma4 as i mentioned, i tried multiple livekit hosted models like gemma, gpt 5.4mini, grok one, i can see timeouts happening for all, it was all working fine before two weeks, so something has changed in livekit hosted models in livekit inference that is causing timeouts?
Just to be clear, the only model we are hosting is gemma4, we aren’t hosting gpt 5.4-mini or grok-4-1, the latter two are just served through Inference.
yeah i mean others are via livekit inference, but intresting thing is if i directly use third party provider via its api key, it works and i have no timeouts, timeouts happen only via livekit inference. I am capturing common graph, so i dont have per model timeout data. so i am using gemma4 only currently and yesterday it still has 22 timeouts. i understand that its in us and not in eu-region, but it should lead to extra delay and still timeouts should not happend.
I need to check with the team, but looking at your server logs for gemma4 and the number of canceled requests, it looks like your project is an outlier compared with the number of similar requests on other projects.
return inference.LLM(
model=f"google/{cfg.model or _DEFAULTS['google']}",
extra_kwargs=extra, # with temperature 0.7 and max_completion_tokens 250
)
Tiemouts are captured live from the Agents SDK event callbacks in our worker:
Errors — error events on each adapter plus the AgentSession “error” event (per your “Events and error handling” doc pattern), recorded as {type, message, recoverable, phase}.
Fallbacks — stt/llm/tts_availability_changed events; we record every transition and whether the fallback ever engaged.
On timeouts specifically — we don’t have a single “timeout” metric; they show up three ways:
LLM attempt timeout — our LLM FallbackAdapter runs with attempt_timeout=2.5s (I was facing same issue with default 5.0. so i reduced as it was causing more latency). A slow primary triggers llm_availability_changed and a switch to fallback, so these appear as fallback engagements.
Provider API timeouts — surface as error events with type APITimeoutError (or provider equivalent); we count them from the recorded error type.
So we have been using livekit from past 2 months, and issue started from 14th july, for 5.4 mini openai model and from last week we have switched to gemma model as it was more faster.
Note that on some days timeouts dropped significantly and almost 0-4, while on other days its much higher, and traffic/sessions are mostly uniform on all these days of july
Thanks, I’m looking into this, can you please share a couple of session IDs (RM_) that use Gemma and show the timeouts, we can then take a deeper look what is happening
Thanks, still investigating this. Hope to provide an update early next week.
# currently primary is gemma4 and secondary is openai5.4 mini
primary = _make_google(
AiConfigLlm(provider="google", model="gemma-4-31b-it"),
custom=False, # false means livekit inference and true means sdk custom using api key
)
fallback = _make_openai(
AiConfigLlm(provider="openai", model="gpt-5.4-mini"),
custom=False, # false means livekit inference and true means sdk custom using api key
)
adapters = [primary, fallback]
return FallbackAdapter(adapters, attempt_timeout=settings.LLM_ATTEMPT_TIMEOUT)
this is how i used it, yea this is not from our side but something from livekit inference internal error, how could it occur only twice than? i mean i am 100% sure i have configured it correctly and nothing is changed from past 2 weeks as i mentioned, i was using openai 5.4 mini before and now gemma4, if it was model not defined issue than from start of the session itself it should be error not just once or twice in between
Also charts i showed is clear proof of it happens for some days while it dont happen much for other days. I have also shared the sessionid, to look into more detail, i think i need to create github issue
Feel free to - I am already speaking with engineering on this, but it can be good to have as tracking also.
What confused me was the Model being FallbackAdapter for those failed spans, as seen in your screenshots. I also see your model specified as FallbackAdapter for your successful spans, so that is consistent. Also, Error getting model definition is confusing, I don’t see how that can happen if you are just specifying google/gemma-4-31b-it which also led me to assume you had multiple models and a fallback.
Actually, to validate my own understanding I set up a new agent with this model and it correctly displays google/gemma-4-31b-it
Yeah so i think if you would try via fallback adapters then maybe you can reproduce that?
Something like below?
# currently primary is gemma4 and secondary is openai5.4 mini
# false means livekit inference and true means sdk custom using api key)
_DEFAULTS: dict[str, str] = {
"openai": "gpt-4o-mini",
"google": "gemini-3.1-flash-lite",
}
def _make_openai(cfg: AiConfigLlm, custom: bool = False) -> llm.LLM:
extra: dict = {}
if not custom:
return inference.LLM(
model=f"openai/{cfg.model or _DEFAULTS['openai']}",
extra_kwargs=extra,
)
# code for custom is after it
def _make_google(cfg: AiConfigLlm, custom: bool = False) -> llm.LLM:
if not custom:
extra: dict = {}
return inference.LLM(
model=f"google/{cfg.model or _DEFAULTS['google']}",
extra_kwargs=extra,
)
# code for custom is after it
primary = _make_google(
AiConfigLlm(provider="google", model="gemma-4-31b-it"),
custom=False
)
fallback = _make_openai(
AiConfigLlm(provider="openai", model="gpt-5.4-mini"),
custom=False
)
adapters = [primary, fallback]
# LLM_ATTEMPT_TIMEOUT used was 2.5
return FallbackAdapter(adapters, attempt_timeout=settings.LLM_ATTEMPT_TIMEOUT)
