We recently started testing with Gemma 4 31B on LiveKit Inference and are noticing a few things:
When it does tool calls and it gets a lot of input back, it hallucinates quite frequently. For example, for a tool task to get appointment slots: which returns slots for Thursday, Friday and Saturday, but somehow the LLM returns Sunday as well…here it hallucinates in our test cases 50% of the time.
We have some complex Agent configs, where Agents have 2 tasks and 1 task group. The tools in the tasks are somehow never called and always missed, even though we pass through the LLM settings (Gemma in this case) to the AgentTask.
In it’s output it sometimes puts “<call” behind a sentence, not sure if this can be handled with prompting.
Are these known issues with Gemma and is there a way to handle these use cases to get better accuracy?
@darryncampbell thanks, but we don’t have Agent Observability enabled, is there specific information required that I can share with support, than I can get it from our logs.
Could it be that larger prompts and context, including that of Tasks and TaskGroups can become confusing for Gemma 4?
Yes, we have the entire trace for the entire conversation, the thing is…we see the agent correctly going into the task, but somehow it doesn’t call the tool at all (referring to tools within the call), and if a tool call has a big response it hallucinates.
Quite possibly something in your complex agent configs that is triggering something… I don’t suppose you are able to share a test harness that reproduces this?
Sometimes Gemma 4 31B through LiveKit inference will fall back to a model from a different provider - it showed as “deepinfra” in the provider label when I checked observability. I had several scenarios where I had the same behaviour you’re describing and it was always when the model response came through that fallback. If you can see the provider in your data then that might help narrow it down.
I’ve had to switch to a different inference provider.
Paul’s lead is worth checking first, and you don’t need Agent Observability to do it. The provider comes through in the metrics metadata, so you can pull it from your own logs.
Each LLMMetrics event carries metadata.model_provider and metadata.model_name, and the same values show up as provider and model on the entries in session.usage. So you can log it per call:
Run one of the conversations where the tool is missed and see what provider is printed. If it says something other than LiveKit on the calls that fail, that matches what Paul saw and it’s a routing problem, not your agent config.
On the tools inside tasks never being called: I’d treat that as a separate issue until the provider question is settled. If it turns out the provider is correct every time, then a minimal repro with one task and one tool would be the fastest way to get someone to look at it.
@darryncampbell We are doing some experimentation and so far we see that simplifying prompt and task instructions makes it a bit better, now all our instructions are tailored to gpt-4.1 mini, where we have 100% accuracy in tool calls
Thanks @vvgr001 , I appreciate the response. Paul’s suggestion above that is is somehow losing context during a fallback sounds sensible, but also not easy to reproduce.
I’ll do my own testing when I get a chance, but in the mean time we’ll keep an eye out for these sort of issues in the future as there’s no smoke without fire
@darryncampbell thank you as well and the support as always. The speed of Gemma on LiveKit Inference is a game changer for us, the model also being in EU (from what I read soon) will also make it even better for many of our use cases.
We have quite complex agents running, where they have 6-8 tools + 4 tasks + 1 task group, so quite a good use case if you want to reproduce on your side. And here we switch between task and main agent quite often during a conversation, so preserving context is key.
@vvgr001 Two of your three symptoms match tracked Inference reports, worth checking before you chase prompting.
The “<call” in the text is a tool-call delimiter leaking into content, the same shape as #6942 (google/gemini-3.5-flash leaks its thought part into delta content with no closing delimiter) (agents#6942). That is Inference-side token handling, so prompting will not fully fix it. Strip it before TTS as an interim:
import re
from typing import AsyncIterable
from livekit.agents import Agent, ModelSettings
class BookingAgent(Agent):
async def tts_node(self, text: AsyncIterable[str], model_settings: ModelSettings):
async def stripped():
async for chunk in text:
yield re.sub(r"<call.*", "", chunk) # drop the leaked token
async for frame in Agent.default.tts_node(self, stripped(), model_settings):
yield frame
Your Tasks/context hypothesis is grounded too: on gemma-4-31b, mid-conversation system messages reach the model as a trailing system role and it answers as the user (agents#7020). Tasks inject exactly those turns, so a multi-task run is more exposed. That fits the extra slot and the confusion inside tasks.
For the task tools never firing, use your trace: check whether the task’s tool schema appears in the LLM request when that task is active. Missing means a registration issue; present but ignored means model selection.
@Muhammad_Usman_Bashir thanks for the feedback…we see basically everything being loaded up correctly in the agent, tools, tasks, tools to initiate tasks etc. etc. However, Gemmma still misses them, so basically task schema’s are registered.