This could be the same issue as reported in Gemma 4 Que Times - LiveKit Inference - #8 by Paul_Barnes
Sometimes Gemma 4 31B through LiveKit inference will fall back to a model from a different provider - it showed as “deepinfra” in the provider label when I checked observability. I had several scenarios where I had the same behaviour you’re describing and it was always when the model response came through that fallback. If you can see the provider in your data then that might help narrow it down.
I’ve had to switch to a different inference provider.