I’ve been using GPT-4o-mini as my LLM for voice agent (inbound and outbound calls). I tried other LLMs a few months ago but either cost, latency, or performance brought me back to 4o.
any suggestions on alternatives tat balance cost and latency with better performance than 4o-mini?
@Robert_hooman Try google/gemma-4-31b-it first. The docs call it "the recommended default LLM... a latency-optimized, open-weight model served on LiveKit's infrastructure", and LiveKit now redirects every deprecated Gemini Flash model to it. It’s tuned for your exact workload: "long system prompts, lots of tools, and a very low latency budget", running on SGLang with speculative decoding.
$1.20/1M output, so more per token than 4o-mini. Whether it’s cheaper overall depends on how output-heavy your calls are. If cost dominates instead, google/gemini-3.5-flash-lite is the other direction.
Worth changing what you benchmark on, though. LiveKit’s framing is the right one: "A voice agent doesn't speak in tokens; it speaks in sentences", so time-to-first-sentence is the number, not TTFT. A model with a fast first token and slow generation still leaves the caller hanging mid-sentence. Test against your real system prompt and tool set, since prompt length and tool count move this more than model choice does.
Hey Abid, thank you for your answer. Yeah I’m hearing this too. I may eventually switch. I switched to gpt-6-luna as the cost is more inline with 4o-mini and gemma-4-31b-it while it seems to be industly standard is running about 2.5 times higher tha 4o-mini and 6-luna is actually cheaper.
@Robert_hooman As Abid has suggested, the google/gemma-4-31b-it is the go-to model that Livekit suggests after it had done benchmark testing.
On my end, I mainly use Realtime models in Livekit, and have found the gemini-3.8-live to be good.