Agent didn't followup after tool call

Agents
,
1

In a certain session, the agent didn’t followup after calling a tool - said it’ll do something, invoked the tool, then didn’t followup. Any attempts to nudge it by saying something resulted in the same behavior happening again - saying it’ll do something, calling the tool, and then not following up.

I couldn’t reproduce this in other sessions, but I suspect it’s likely to happen again, even if rarely.

Any idea why this happened?

I’m using xAI Realtime with the Node SDK.

Agent id: A_9ZooAu8swT7M

Session id: RM_Z3H9LfYTWbzT

2

@royibernthal
This is likely the behavioral difference between xAI Realtime and OpenAI Realtime in how post-tool turns are triggered.

With xAI (and other non-OpenAI realtime models), LiveKit swaps tools for the reply then restores them, but that restoration doesn’t automatically trigger a follow-up turn the way OpenAI Realtime does. Grok just goes silent if it doesn’t self-initiate.

Two things to try: explicitly call generate_reply() after your tool returns, and check max_tool_steps. If the limit is hit and the final LLM call produces no audio with xAI, you’d see exactly this pattern.

The inconsistency across sessions points to model-side behavior rather than your code.

3

We won’t have any helpful information in our logs as we do not log your agents logs.

Have a look at your agent logs to see what may have happened:

Also it maybe helpful to check Agent Insights too if you have that enabled

4

@royibernthal, the agents framework handles the multi-step case explicitly: after tool execution, if num_steps >= max_tool_steps + 1, it forces tool_choice="none" on the follow-up call to “guarantee a final text response instead of silently stopping” [ agent_activity.py:2855-2914 ].

xAI Realtime under that tool_choice="none" constraint can return text without producing audio. That fits your symptom: tool fires, then silence. The intermittency would line up with whether the specific turn happens to hit max_steps_reached or draining state.

Practical implications can be: raise max_tool_steps if your tool chains are deeper than the default, and add a defensive watchdog on the post-tool turn:

  session.on(voice.AgentSessionEventTypes.MetricsCollected, (ev) => {
    // if the post-tool turn produced no audio output, nudge once
    if (ev.metrics.outputTokens === 0) {
      session.generateReply()
    }
  })

That same handler doubles as confirmation if you’re trying to reproduce, outputTokens=0 on the post-tool RealtimeModelMetrics proves the empty-audio path.

5

Sorry for the late response, had a lot going on.

@CWilson I could share the logs. How can I export to a file via the CLI the logs of a session that already happened? Clicking on “Download data” at the top right in the session page in the dashboard doesn’t export the logs.

@khan @Muhammad_Usman_Bashir

Thanks for the explanations.

Sounds like a bug that might be worth fixing in the livekit repos?
I see a similar one has already been fixed:

ev.metrics only contain timestamp and type in the node sdk, I don’t see outputTokens.

I don’t see in the session logs anything regarding outputTokens, max_steps_reached, or max_tool_steps. Maybe these are not logged by default?

I do see these logs mentioning interruptions:
playout interrupted
playout completed with interrupt
Aborting all realtime generation tasks due to interruption

It’s worth noting that I haven’t been able to reproduce the issue since opening this topic. I only have the problematic session.

6

@royibernthal, Good find on #4747, that’s the fix I was pointing at, so the max_tool_steps silent-stop is already closed. No max_tool_steps logs in your session means that’s not it. Scratch that, and my outputTokens watchdog too; the Node MetricsCollected payload doesn’t carry it.

The interruption logs are the lead. “Aborting all realtime generation tasks due to interruption” at the tool boundary means the post-tool reply started, got interrupted, and the realtime generation was aborted instead of spoken. The framework auto-recovers from this via turnHandling.interruption.falseInterruptionTimeout (default 2000ms) + resumeFalseInterruption (default true) [ agents-js turn_config/interruption.ts ], but resume is gated on a pausable output (pauseEnabled() checks output.audio.canPause) [ agents-js agent_activity.ts ], and an aborted realtime generation may not take that path. So it stays silent and re-nudging repeats it. Likely the bug you’re sensing.

Node hook and stopgap:

session.on(AgentSessionEventTypes.AgentFalseInterruption, (ev) => {
  if (!ev.resumed) session.generateReply();   // didn't recover, nudge once
});

If you file it, the repro is the interrupt firing right after the tool returns with no preceding user transcript.

7

Got it, I’ll try that solution and keep you updated if I manage to repro it. Thanks!