LiveKit Inference timeouts happening since past 2 weeks

[Kaushal_Shah]

@CWilson @Adrian_Cowham Thanks, we can mark this as solved! The timeout errors are solved.

However, I’m seeing high TTFS and TTFT than before for gemma. Looking at benchmarks, the TTFT seems to be getting progressively higher each day. I was expecting consistently fast performance, is there a way to optimize this, or should I open a new post to discuss it?

[Adrian_Cowham]

Hey Kaushal,

Thanks for the update! The increased latency is due to scaling on our end. We’re adding new providers for gemma and incrementally sharing the load. Optimizing these new providers is an ongoing endeavor, we hope to have things ironed out this week.