@Kamal_Moha@abidullahcs.uk has this right, and the source confirms it. with_filler’s loop starts each pass with await session.wait_for_idle() (filler_scheduler.py#L70-L78), and wait_for_idle stays busy while a speech handle is current, clearing only when _current_speech is None (agent_activity.py#L1929-L1938). A blocking tool holds that speech current, so the filler loop parks at line one. That is Scenario 2. Your ctx.update() releases the block by flagging the call non-blocking, so Scenario 1 fires fillers but defers the result.
So yes, mutually exclusive with the built-in with_filler today, and worth filing as a docs-versus-behavior gap: it is documented for filler speech while a long-running step blocks the tool, yet its loop needs the idle a blocking step never yields. Your two scenarios are the repro.
To get both now, keep the tool blocking so the return is spoken normally, and drive the progress speech yourself:
async def _progress(session, lines, first=2.0, gap=12.0):
await asyncio.sleep(first)
for line in lines:
session.say(line)
await asyncio.sleep(gap)
@function_tool
async def track_package(self, ctx: RunContext, tracking_num: str) -> str:
task = asyncio.create_task(_progress(ctx.session, ["Still tracking.", "Almost done."]))
try:
result = await wait_for_redis_result(tracking_num) # tool stays blocking
finally:
task.cancel()
return f"Tracking {tracking_num}: {result}" # spoken as the tool output
No ctx.update(), so nothing flips the call non-blocking and the return stays the real output. Keep gap generous, since you lose with_filler’s talk-over guard here.
Hi @abidullahcs.uk@Muhammad_Usman_Bashir thanks for the guidance on how can I do the task of updating the user while tool is being executed. I have updated my code to keep updating the agent. Here is the track_package tool
class TrackOrder:
def __init__(
self, phone_number: str, room_sid: str, room_name: str, job_id: str
):
self.phone_number = phone_number
self.room_sid = room_sid
self.room_name = room_name
self.job_id = job_id
@function_tool(on_duplicate="reject")
async def track_package(
self,
ctx: RunContext,
tracking_num: str,
) -> str:
"""Track a shipment/package for the user.
Args:
tracking_num: Tracking number to check
"""
request_id = f"{self.room_sid}-{tracking_num}-{self.job_id}-{self.room_name}"
channel_name = f"shipment:done:{request_id}"
result_key = f"shipment:result:{request_id}"
async def _progress(ctx, lines, first=2.0, gap=12.0):
await asyncio.sleep(first)
for line in lines:
async with ctx.foreground():
await ctx.session.say(line)
await asyncio.sleep(gap)
# ignore_subscribe_messages=True prevents Redis setup frames from entering .listen()
pubsub = redis_client.pubsub(ignore_subscribe_messages=True)
task = asyncio.create_task(
_progress(
ctx,
["Still tracking.", "Almost done."]
)
)
try:
# # 1. Subscribe to Redis channel before triggering background worker
await pubsub.subscribe(channel_name)
print(f"Subscribed to channel: {channel_name}")
# 2. Trigger Inngest Event passing Redis routing metadata
await inngest_client.send(
inngest.Event(
id=f"{self.phone_number}-{request_id}",
name="shipment/track.package",
data={
"tracking_num": tracking_num,
"phone_number": self.phone_number,
"request_id": request_id,
"channel_name": channel_name,
"result_key": result_key,
},
)
)
async def wait_for_redis_push() -> str:
async for message in pubsub.listen():
print(f"Message: {message}")
if message and message.get("type") == "message":
return str(message["data"])
return ""
# Wait for Redis push with a 120s timeout
message_data = await asyncio.wait_for(
wait_for_redis_push(),
timeout=120.0,
)
# Fetch completed result or fallback to message payload
result_data = await redis_client.get(result_key)
final_status = result_data or message_data
print(f"FINAL Status: {final_status}")
parsed = json.loads(final_status)
summary = parsed.get("summary")
issues = parsed.get("issues")
origin = parsed.get("origin")
destination = parsed.get("destination")
timeline = parsed.get("timeline")
carrier = parsed.get("carrier")
return (
f"""
Tracking update for {tracking_num} is:
Summary: {summary}
Issues: {issues}
Carrier: {carrier}
Origin: {origin}
Destination: {destination}
Timeline: {timeline}
"""
)
except asyncio.TimeoutError:
return (
f"Tracking for {tracking_num} is taking longer than expected. "
"I will update you as soon as the system completes it."
)
except Exception as e:
print(f"Tracking tool error: {e}")
return (
"There was an error checking your shipment. Please try again."
)
finally:
task.cancel()
# Cleanup PubSub connection and local locks
await pubsub.unsubscribe(channel_name)
await pubsub.aclose()
return (
f"""
Tracking update for {tracking_num} is:
Summary: {summary}
Issues: {issues}
Carrier: {carrier}
Origin: {origin}
Destination: {destination}
Timeline: {timeline}
"""
)
...
I have tested it, but I see like what I had in scenario 2. The agent remains quiet while the tool is being executed and once the tool output is returned, the agent talks to the user. This is what I have noticed when I implemented this.
@Kamal_Moha Both changes you made to that snippet put a wait back in.
ctx.foreground() is the bigger one. Its docstring says “Wait for idle, then hold the floor”, and the implementation is literally activity = await self.wait_for_idle() before it yields. So wrapping say() in it restores the exact gate you were trying to step around.
The second is await ctx.session.say(line). say() is a synchronous method that returns a SpeechHandle, so calling it queues the speech and returns straight away. Awaiting the handle waits for full playout instead.
Run it without either:
for line in lines:
ctx.session.say(line)
await asyncio.sleep(gap)
Worth adding either way, since a bare create_task swallows exceptions and silence looks identical to a crash:
def _died(t: asyncio.Task) -> None:
if not t.cancelled() and t.exception():
print("progress task died:", t.exception())
task.add_done_callback(_died)
I have switched to using the STT-LLM-TTS pipeline and I’m seeing that ctx.with_filler() is performing better with the pipeline than with Realtime models.
ctx.with_filler() only speaks through session.say(). So when using the realtime model, ctx.with_filler() doesn’t generate speech because it doesn’t use say()