I’ve seen similar results when using Gemma 4 31B through LiveKit inference. From the session logs the model seems to fall back to deepinfra as the inference provider, which results in higher latency as well as considerably worse output - there seems to be an issue with the checkpoint that deepinfra is serving, I’ve seen several issues with the model claiming a tool call that wasn’t called or even duplicating its response like here:
Likely offloading to deepinfra due to increased demand as already mentioned.
The LLM Inference class does have a parameter to allow specifying a provider, @darryncampbell do you know how we can set this to only get the LiveKit-hosted checkpoint and avoid the fallback? I’d much rather be able to control the fallback myself through a FallbackAdapter.
Love the model when it doesn’t fall back - and I was planning to move my production deployment to the LiveKit Cloud Ship plan to get reliable access to it - but without being able to prevent it falling back like this it’s essentially unusable.