Will it ever be considered to add the ability for people to load LoRAs onto Gemma 4 via LiveKit Inference?
I haven’t seen it discussed previously but I did ask @Neil_Dwyer
Thanks for asking for me. I think LoRA support (through multi-LoRA) would be really valuable for voice agents.
Being able to fine-tune a good base model like Gemma 4 31B for the specific conversational behavior, tool usage, and reliability patterns helps cut out a lot of that from having to exist in a massive universal prompt (which brings improved latency) while making agents more capable and much more natural.
Something I’ve found from working on voice agents is that there’s a limit to how much behavioral consistency you can get from the universal prompt alone. You end up encoding the same desired behavior through increasingly detailed instructions and lots of examples just to reliably steer a model whose underlying weights weren’t specifically trained for that behavior.
I’d imagine there’s a benefit in it for LiveKit’s too. Smaller per-session context meaning less KV-cache memory/compute, particularly when you have a lot of concurrent inference going on.
Always would give LiveKit yet another edge, because no other providers are doing it for Gemma 4, but since you guys have enough concurrent people using these small opensource models it could be worth pursuing. Multi-LoRA seems like a good idea to look into supporting.
Would definitely be something that would cause me to switch a ton of my usage over to you guys.
So you are asking if we would run GPUs to host your own fine-tuned models instead of a common shared model? Curious why you want to use LoRA instead of a full scale fine tuned model? Are you worried about memory size?
My thesis was that LoRA would be much more accessible to people who rely on LiveKit Inference, while getting you most of the way there for the kinds of things I’m looking to optimize. In my case, I’d want to train it on our own client calls to improve conversational behavior, tool usage, reliability, and overall agent quality and naturalness.
The reason I brought it up is that I think LoRA has a lot of potential for voice agents specifically. Everyone is building agents for different industries and use cases, and everyone has slightly different ideas of what ideal agent behavior looks like. A full fine-tune of the base model could obviously improve its general voice-agent performance, but it can’t optimize for every niche use case or individual developer’s preferences.
My ideal would be for LiveKit Inference to eventually let us load our own LoRA adapters, potentially using multi-LoRA through SGLang. That would let LiveKit keep serving the same optimized base models while developers customize them for their own agents without requiring a completely separate model deployment.
That seems like a relatively efficient way to unlock a pretty significant increase in customization and agent quality.
As @darryncampbell mentioned above, he has brought it to the team’s attention. I was just curious about your reasoning.
Thanks for the extra details.
Of course, would happy to answer any additional questions and help in any way possible. Feel free to just message me on LiveKit or send me an email.
Hi, I was just curious, would I be able to get an answer once you hear something back from the team? This decision would somewhat effect the direction of my business, so it would be extremely helpful to know.
I can let you know when I hear something back. But my guess is this probably will not will be be available in the near future.
Ah that’s okay, thanks for your help. Unfortunate because I think it’s a really impactful idea that will seem obvious to have for voice agents in hindsight.
And so it begins. One of the first major full-weight fine tunes. But even with full-weights fine tune, a LoRA is still extremely helpful for optimizing that last bit to make an agent feel extremely knowledgable, and natural within specific industries, without having to carry a bloated system prompt.
Even if the team doesn’t think it’s worth it, or viable to support LoRA with Gemma 4 31B, I would ask you look into hosting PhoneLLM Alpha 1 as the fine tune seems pretty good in my limited testing, and then potentially allowing us to load a LoRA with that. It’s a much smaller model (only 3B active params), so should have an easier time serving it too.
Would love to hear your thoughts and the teams. @darryncampbell @CWilson
Thanks for sharing, it’s definitely an exciting time to be in the industry.
I wrote a whole paragraph but then decided not to post it in reply here… I’m honestly limited in what we can share publicly right now but the team are not sitting still