We are using GPT 4.1 and later OpenAI models (including the OSS models) the most as those seem to have the most accurate tool calling capabilities.
Somehow; within LiveKit agents, it seems like no other capable model, which scores good in benchmarks and in posts for tool calling, works well with tool calling when you put it to the test in LiveKit Agents. Some examples:
Nemotron 3 Ultra & Super: these two models score 90%+ and even 100% when it comes to tool call capabilities. But when you try it or enable them, they fall flat and nothing happens and no tools are invoked. I read somewhere that this might be the case, because of the reasoning and thinking effort of this models, which somehow prevents it from doing tool call correctly and that needs to be disabled? Which is weird. How is this possible and can anyone perhaps provide some best practices?
We noticed that Nemotron 3 models are really good in conversational settings and approaches, but fail 100% of the time in tool calls in LiveKit.
We have used this exact model on AWS Bedrock Mantle, and it scores terrible on tool calls. To give an example: we have a tool called “finish_conversation”, which the agent calls as soon as it detects the user is done or the call is finished.
Gemma 4 31B, literally each and every time, just outputs this as a chat message:
<call:finish_conversation farewell_message=“Thank you for calling and have a nice day!”/>
The results we have been getting with Gemma 4 31B is 0% accuracy in tool calls.
What is the issue or challenge here and what might I be missing? Is this something with settings or configuration that I might be missing? Would be great to know from someone in LiveKit team. Thanks in advance.
All of this is within a STT>LLM>TTS pipeline of course.
What you’re seeing usually means the model is not actually returning a structured function call, but plain text that looks like one. LiveKit Agents only invokes tools when the LLM uses the provider’s native tool/function-calling API — not when it emits XML-like tags in chat.
Key points:
Tools must be defined with a valid JSON object schema and sent through the provider’s function-calling interface, as described in Function tools.
When using non-OpenAI models via the OpenAI-compatible plugin, they must truly support the OpenAI Chat Completions function-calling format — not just “simulate” it in text. See OpenAI-compatible LLMs.
If Gemma outputs <call:finish_conversation .../>, that indicates the model isn’t using structured tool calls at all — it’s formatting text.
@CWilson we indeed use the @function_toolfunction_tool approach with all our tools, as prescribed by the livekit documentation.
We load tools dynamically through database configuration, but they are given the @function_tool decorator and have the same structure as the lookup weather function from the documentation. Or must we add the raw_schema as well?
Also this does work with OpenAi models and open oss 120b models on groq, cerebras and aws.
Not sure why Gemba 4 31B is giving raw text output in this case. Or did I miss something?
Or as you mentioned; this might have to do with the specific model deployed on a specific provider?
No, you didn’t miss anything on the tool-definition side — @function_tool is correct and raw_schema won’t change anything here. In fact, your Gemma output proves the schema is arriving fine: the model emitted your exact tool name and a well-formed farewell_message argument. It saw the tool and decided to call it. The failure is entirely on the response path — how the provider serializes that decision back to you.
And yes, your last sentence is exactly it: it’s per (model + provider) pairing, not per model.
Here’s why your working cases work: OpenAI models emit the OpenAI tool-call format natively. gpt-oss-120b emits its own “harmony” format, but Groq, Cerebras, and Bedrock all ship a parser for it, so their API layer converts it into structured tool_calls before it reaches you. Gemma emits yet another format (the <call:...> tag you’re seeing), and the endpoint you’re using apparently has no parser for that format — so the tag falls through as plain assistant text. LiveKit Agents only reads the structured tool_calls field (by design — text-scraping tool calls is unreliable), so from the framework’s perspective the model returned an ordinary chat message, and it gets spoken.
You can prove it in one curl, no LiveKit involved — call your Bedrock endpoint directly with a tools array and a prompt that should trigger a call:
Call lands in message.tool_calls → serving is fine, and I’d want to look deeper at your agent config.
Call text lands in message.content → the provider isn’t parsing Gemma’s format, and nothing in LiveKit Agents (raw_schema included) can fix it.
I’d bet on the second outcome. If Gemma is a model you want, your paths are: LiveKit Inference (we run the Gemma tool-call parser gateway-side — that’s what the accuracy number on the product page is measured against), a self-hosted stack with the Gemma tool parser configured (e.g. vLLM’s --tool-call-parser), or checking whether Bedrock’s native Converse API returns structured toolUse for Gemma even where the OpenAI-compat layer doesn’t.
@vvgr001 are you using the node SDK or the python one?
I’ve been playing around with the node SDK myself, and ran into this bug with the SDK using the Gemma model. I’ve traced it through and looks like there is an unresolved bug in the gateway. Here’s Claude’s take:
Root cause found for the “Gemma silently doesn’t call tools” issue — it’s a $schema key in the tool params, not the model or a missing gateway parser.
We hit this with google/gemma-4-31b-it on LiveKit Inference and traced it end to end. Contrary to what you’d assume, Gemma does return proper structured tool_calls on LiveKit Inference — the gateway’s Gemma tool parser works fine. The failure is caused by a single field in the outgoing request. When you define a tool with a Zod schema (llm.tool({ parameters: z.object(...) })), the JS SDK converts it to JSON Schema via the zod-to-json-schema library, which stamps a top-level "$schema": "https://json-schema.org/draft/2019-09/schema#" onto the tool’s parameters. That value makes the gateway route the request off its Gemma tool-parsing deployment onto a fallback deployment that doesn’t parse tool calls — so the stream just ends with no tool_calls and finish_reason: undefined, and the framework speaks an empty/normal message. We confirmed it’s the value, not the mere presence of the key: sending $schema: http://json-schema.org/draft-07/schema# works fine, while the draft/2019-09 value (and any unrecognized value) breaks it. A hand-rolled raw request without the key returns the tool call every time; adding just that one field flips it to failure, deterministically.
Two useful notes. First, LiveKit has already fixed this upstream — on the agents-jsmain branch, src/inference/llm.ts now strips the key (const { $schema: _dropped, ...parameters } = llm.toJsonSchema(...)), with a comment describing this exact behavior. It’s not in a released version yet (we’re on 1.5.0, which still passes $schema through), so if you’re on npm latest you’ll still hit it. Second, zod-to-json-schema has no option to suppress $schema — it’s set unconditionally based on target. So until the upstream fix ships, the local workaround is to strip the key yourself before the request (mirror LiveKit’s approach: destructure $schema out of the generated schema rather than delete, so a raw-JSON-schema caller isn’t mutated). Alternatively, tools defined with Zod v4 emit the tolerated draft-07 value and work as-is — but that relies on the gateway’s draft whitelist, so stripping the key is the more robust fix.
Worth flagging for anyone starting from the official template: livekit-examples/agent-starter-node ships with Zod v3, which means tool calls do not work out of the box with Gemma because of this exact issue — the v3 → JSON-Schema conversion embeds the draft/2019-09$schema value that Gemma’s gateway deployment rejects. You won’t see an error; the tool just silently never fires. To get it working you either have to (a) switch to a model whose backend tolerates the extra key (e.g. Gemini or an OpenAI id — note this “works” only because those deployments ignore the bad $schema value, not because the SDK sends anything different for them), or (b) strip the $schema key from your tool params before the request. Until the upstream fix on main is released, option (b) is the reliable path if you specifically want Gemma.