Agent Session Say Done Callback

Server SDKs
, , ,
1

Hi everyone,

As I explained in my last question, I’ve implemented a custom behaviour in my python server where I handle a text input from my client, then I make a request to another server (of my own) and this response I send it back to the client using AgentSession.say() to generate a video and audio response with LiveAvatar. I noticed that if I add a done_callback it just never get trigger.

This is my current implementation:

handle = await session.say(respuestaChatBot, allow_interruptions=True)
handle.add_done_callback(lambda _: print("Se ha acabado la frase"))

I’ve also tried this way:

handle = await session.say(respuestaChatBot, allow_interruptions=True)
handle.add_done_callback(lambda _: print("Se ha acabado la frase"))

And this other way:

async def safe_way(text: str):

            await session.interrupt() #If I've another speaking running
            session.say(text, allow_interruptions=True)

asyncio.create_task(safe_way(respuestaChatBot))

All this code runs inside custom_text_input_handler_tts method. Thank again for your time!

2

Have you tried this way?

handle = session.say(respuestaChatBot, allow_interruptions=True)

handle.add_done_callback(lambda _: print(“Se ha acabado la frase”))

await handle

3

Hi C. Wilson, yes I’ve also tried but it’s still the same. Additionally, it seems to trigger once the agent get disconnected from the session.

4

Hi, had you found a way to get it work? I’m still at the same point from my last post.

Thank again for your work.

5

I don’t think I am understanding the issue you are having.

6

@Antonio_Palomino adding a more complete code sample would help, ideally based on GitHub - livekit-examples/agent-starter-python: A complete voice AI starter for LiveKit Agents with Python. with minimal changes to show the issue. That also helps identify if it’s something unique with your setup

7

Okay, my code now structures this way:

Defining the agent using the @server.rtc_session() decorator:

ctx.log_context_fields = {
 "room": ctx.room.name,
}
# Set up a voice AI pipeline using OpenAI, Cartesia, AssemblyAI, and the LiveKit turn detector
session = AgentSession(        
  tts=elevenlabs.TTS(
    model="eleven_flash_v2_5", voice_id="ERYLdjEaddaiN9sDjaMX", voice_settings=setVoices,    language="es"
   ),
  userdata={}
)

avatar = liveavatar.AvatarSession(
       avatar_id=os.getenv('LIVEAVATAR_AVATAR_ID'),  # See https://docs.livekit.io/agents/models/avatar/plugins/hedra
     )
    
# # Start the avatar and wait for it to join
await avatar.start(session, room=ctx.room)
     
# Start the session, which initializes the voice pipeline and warms up the models
await session.start(
  agent=Assistant(),
  room=ctx.room,
  room_options=room_io.RoomOptions(
    text_input=room_io.TextInputOptions(text_input_cb=custom_text_input_handler_tts)
   ),
 )

await ctx.connect()

Then the function used to detect when the user send an input:

async def custom_text_input_handler_tts(session: AgentSession, event: room_io.TextInputEvent) -> None:
    
    session.interrupt() #Implemented to stop manually the previous session.say if not finished correctly
    try:
        payload_bytes = event.text
        session._update_agent_state("thinking") #Update state manually cause with session.say() don't get update
        
        #If message is not void then proceed with my implementation
        if payload_bytes != "" or not(not payload_bytes):
            #Choose a random waiting phrase
            waiting_phrase = handle_waiting_phrases()
            handle_waiting_phrase = session.say(waiting_phrase, allow_interruptions=True)
            handle_waiting_phrase.add_done_callback(lambda _: print("Se ha acabado la frase"))
            

            #Ask to my own server about a response from the user Input
            respuestaChatBot = await askChatbot(userMessage)

            #Generate the video and audio from these response
            handle_response = session.say(respuestaChatBot, allow_interruptions=True)
            handle_response.add_done_callback(lambda _: print("Se ha acabado la frase de respuestaChatBot"))
             session._update_agent_state("listening")#Update state to start speaking from fronted again
             await handle_response #This way it never trigger the add_done_callback

      except Exception as e:
        text_to_speak = "Lo siento, pero no he podido procesar correctamente la solicitud. ¡Inténtalo de nuevo!"
        session.say(text_to_speak, allow_interruptions=True)
        session._update_agent_state("listening")#Update state to start speaking from fronted again

I simplified a little in this function to focus on the session.say() implementation and how it nevers trigger the add_done_callback event.

Thank again for your time!

8

I’m concerned b y your calls to _update_agent_state, that is an internal function (indicated by the _) so I would not recommend calling it in your agent - the only public interfaces for the agent state are for retrieval.

You might need to rework how you handle agent state, you might find these resources useful: Agent state | LiveKit Documentation or Agent speech and audio | LiveKit Documentation

1 Like
9

I understand, I’ll try to handle using these resources. Thanks!!