Chat context contains text that wasn't spoken due to interruption

Agents
1

This question originally came up in our Slack community and the thread has been consolidated here for long-term reference.

When the user interrupts the agent before it starts speaking, the chat context still contains text that was streamed from the LLM but never spoken by TTS.

Is there a way to make sure only the text that was actually spoken goes into the chat context?

2

LiveKit has a TranscriptSynchronizer component that estimates how much text has been played out. When an interruption occurs, it should add only the text that was heard by the user to the chat history.

However, this estimation isn’t perfect - it can be off by a word or two, especially for non-English languages.

To handle interrupted items, you can use the conversation_item_added event to modify the context:

@session.on("conversation_item_added")
def on_conversation_item_added(item: ConversationItemAddedEvent):
    if (isinstance(item.item, ChatMessage) and 
        item.item.role == "assistant"):
        
        if hasattr(item.item, 'interrupted') and item.item.interrupted:
            # Remove the interrupted item from context
            current_ctx = agent.chat_ctx
            filtered_ctx = ChatContext()
            
            for ctx_item in current_ctx.items:
                if ctx_item.id != item.item.id:
                    filtered_ctx.items.append(ctx_item)
            
            asyncio.create_task(agent.update_chat_ctx(filtered_ctx))