Publisher pc state failed and the agent resumed frequently at around the 8-minute mark

Agents
,
1

We’ve been consistently seeing errors on longer calls, usually around the 8-minute mark, such as:

livekit::rtc_engine::rtc_session:1161:livekit::rtc_engine::rtc_session - Subscriber pc state failed
livekit::rtc_engine:780:livekit::rtc_engine - resuming connection... attempt: 0

After the connection resumes, the previous TTS publication track settings are lost, which causes unexpected audio behavior from the agent. For example, if the agent was muted before the reconnect, it becomes unmuted afterward.

This issue has become much more consistent over the past 12 hours and is now occurring on essentially every call that lasts 7–8 minutes.

2

Hi, just looking internally, I don’t see any other reports of this.
Can you give me a few session IDs (starts with RM_) and I’ll check the server logs.

3

Hi, these are some session IDs: RM_zLM3XZQTsfkx, RM_R7G4aTZrxxYS

4

It looks like the connection issues are coming from the agent ↔ LiveKit leg rather than client ↔ LiveKit leg. I notice your agent is self-hosted, did anything change in your agent hosting environment? I don’t see any obvious root cause

5

We haven’t changed any code in our agent for the past two weeks. This issue only started occurring in the last couple of days, especially during long calls.

Has LiveKit made any recent changes related to timeout configuration or anything similar? In our case, we may mute the agent for a long period of time during a call.

6

I could reproduce the issue with sample Livekit code on my computer (not from a server).

Here’s the error:

    01:14:24.298 ERROR    livekit            livekit::rtc_engine::rtc_session:1292:livekit::rtc_engine::rtc_session - Publisher pc state failed  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
                 WARNING  livekit            livekit::rtc_engine:503:livekit::rtc_engine - received session close: "pc_state failed" UnknownReason Resume  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
                 ERROR    livekit            livekit::rtc_engine:813:livekit::rtc_engine - resuming connection... attempt: 0  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:24.533 WARNING  livekit            livekit::rtc_engine:503:livekit::rtc_engine - received session close: "signal client closed: \"stream closed\"" UnknownReason Resume  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
7

I am not seeing the issue. Can you please reproduce the issue with basic_agent.py and provide detailed steps for how we can reproduce the issue?

Also if you can run “pip freeze |grep livekit" and also include what OS you are running and a full log from the agent. Maybe we will see what the issue is. I expect there is something wrong with your agent code somewhere.

8

Fair, @Quoc_Ngo’s reproducer is the genuine finding that weakens @CWilson’s “agent code is wrong” framing, and there’s a testable hypothesis OP can run while waiting on the data @CWilson asked for.

9

I had to make a 8 minute long so there’s a lot of log. Pasting only chunks around the error time here:

01:13:03.657 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.12975268065929413, "duration": 0.061, "input": "<|im_start|>assistant\nlinen prices can vary widely and buying from 
manufacturers in china can sometimes be more affordable however quality can differ so it's important to check reviews and materials would you like help finding options 
online<|im_end|>\n<|im_start|>user\nabsolutely can you help me find something online<|im_end|>\n<|im_start|>assistant\nsure can you tell me your preferred size and any specific 
styles or colors you have in mind for the linen shirt<|im_end|>\n<|im_start|>user\nm size medium size<|im_end|>\n<|im_start|>assistant\ngot it medium size do you have a preferred 
color or style for the linen shirt like a casual look or something more formal<|im_end|>\n<|im_start|>user\ncasual note but doesn't have to be you know super casual", "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:03.937 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "05f6e5611deb", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:06.813 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "That's", "language": "en", "transcript_delay": 0.020261049270629883, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:07.775 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "to", "language": "en", "transcript_delay": 0.03100895881652832, "room": "console-c59e1513", "pid": 86422, "job_id": 
"AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:08.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:08.781 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "know,", "language": "en", "transcript_delay": 0.03646492958068848, "room": "console-c59e1513", "pid": 86422, "job_id": 
"AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:10.846 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "others.", "language": "en", "transcript_delay": 0.45096707344055176, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:11.018 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.023826871067285538, "duration": 0.07, "input": "<|im_start|>user\nabsolutely can you help me find something 
online<|im_end|>\n<|im_start|>assistant\nsure can you tell me your preferred size and any specific styles or colors you have in mind for the linen 
shirt<|im_end|>\n<|im_start|>user\nm size medium size<|im_end|>\n<|im_start|>assistant\ngot it medium size do you have a preferred color or style for the linen shirt like a casual 
look or something more formal<|im_end|>\n<|im_start|>user\ncasual note but doesn't have to be you know super casual that's to know others", "room": "console-c59e1513", "pid": 86422,
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
                 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 0.17171096801757812, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:13:11.293 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "9fb043df3bc9", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:13.531 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:18.530 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:23.526 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:27.243 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Color do I have?", "language": "en", "transcript_delay": 0.5475902557373047, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:27.315 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.0033080698922276497, "duration": 0.062, "input": "<|im_start|>user\nm size medium 
size<|im_end|>\n<|im_start|>assistant\ngot it medium size do you have a preferred color or style for the linen shirt like a casual look or something more 
formal<|im_end|>\n<|im_start|>user\ncasual note but doesn't have to be you know super casual that's to know others<|im_end|>\n<|im_start|>assistant\nunderstood a medium-sized linen 
shirt in a casual style sounds great ill find options that are versatile enough for different occasions would you like me to focus on specific colors or should i look for a 
variety<|im_end|>\n<|im_start|>user\ncolor do i have", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:28.533 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:29.696 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 2.45255708694458, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:13:29.976 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "a9cf8afefe75", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:33.529 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:38.530 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:41.313 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Suitable for", "language": "en", "transcript_delay": 0.017598867416381836, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:43.526 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:45.302 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "any color, or does it have any color limitation?", "language": "en", "transcript_delay": 0.45787477493286133, "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:45.482 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.7766729593276978, "duration": 0.084, "input": "<|im_start|>user\ncasual note but doesn't have to be you know super 
casual that's to know others<|im_end|>\n<|im_start|>assistant\nunderstood a medium-sized linen shirt in a casual style sounds great ill find options that are versatile enough for 
different occasions would you like me to focus on specific colors or should i look for a variety<|im_end|>\n<|im_start|>user\ncolor do i have<|im_end|>\n<|im_start|>assistant\ndo 
you have any favorite colors in mind like white blue or something else<|im_end|>\n<|im_start|>user\nsuitable for any color or does it have any color limitation", "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:45.483 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 0.18008708953857422, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:13:45.758 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "c1c474f330c2", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:48.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:53.532 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:13:58.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:03.529 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:08.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:09.956 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Oh, I forgot to mention is, uh, for my wife.", "language": "en", "transcript_delay": 0.21146202087402344, "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:10.152 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Not for me.", "language": "en", "transcript_delay": 0.4071769714355469, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:10.372 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.31402072310447693, "duration": 0.076, "input": "<|im_start|>assistant\nunderstood a medium-sized linen shirt in a 
casual style sounds great ill find options that are versatile enough for different occasions would you like me to focus on specific colors or should i look for a 
variety<|im_end|>\n<|im_start|>user\ncolor do i have<|im_end|>\n<|im_start|>assistant\ndo you have any favorite colors in mind like white blue or something 
else<|im_end|>\n<|im_start|>user\nsuitable for any color or does it have any color limitation<|im_end|>\n<|im_start|>assistant\nlinen shirts come in a wide range of colors so you 
can find versatile options that match well with many outfits classic colors like white light blue or neutral shades like beige and gray are often great choices would you like to 
explore those options<|im_end|>\n<|im_start|>user\noh i forgot to mention is uh for my wife not for me", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", 
"room_id": "RM_BAqYvEiXs8ji"}
    01:14:10.373 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 0.2201371192932129, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:14:10.647 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "594b7a2307d2", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:13.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:18.530 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:23.503 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Can we have a ping one?", "language": "en", "transcript_delay": 0.5086910724639893, "room": "console-c59e1513", "pid": 
86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:23.529 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:23.609 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.24410401284694672, "duration": 0.063, "input": "<|im_start|>assistant\ndo you have any favorite colors in mind like 
white blue or something else<|im_end|>\n<|im_start|>user\nsuitable for any color or does it have any color limitation<|im_end|>\n<|im_start|>assistant\nlinen shirts come in a wide 
range of colors so you can find versatile options that match well with many outfits classic colors like white light blue or neutral shades like beige and gray are often great 
choices would you like to explore those options<|im_end|>\n<|im_start|>user\noh i forgot to mention is uh for my wife not for me<|im_end|>\n<|im_start|>assistant\nno problem at all 
so we're looking for a medium-sized linen shirt for your wife are there specific colors or styles she prefers<|im_end|>\n<|im_start|>user\ncan we have a ping one", "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:23.610 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 0.10621523857116699, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:14:23.883 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "213664e4e0a1", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:24.298 ERROR    livekit            livekit::rtc_engine::rtc_session:1292:livekit::rtc_engine::rtc_session - Publisher pc state failed  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
                 WARNING  livekit            livekit::rtc_engine:503:livekit::rtc_engine - received session close: "pc_state failed" UnknownReason Resume  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
                 ERROR    livekit            livekit::rtc_engine:813:livekit::rtc_engine - resuming connection... attempt: 0  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:24.533 WARNING  livekit            livekit::rtc_engine:503:livekit::rtc_engine - received session close: "signal client closed: \"stream closed\"" UnknownReason Resume  
                                  {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:28.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:33.525 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:38.529 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:43.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:46.035 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Yeah. Medium size, please.", "language": "en", "transcript_delay": 0.6894328594207764, "room": "console-c59e1513", 
"pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:46.146 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.8573863506317139, "duration": 0.105, "input": "<|im_start|>assistant\nlinen shirts come in a wide range of colors so
you can find versatile options that match well with many outfits classic colors like white light blue or neutral shades like beige and gray are often great choices would you like to
explore those options<|im_end|>\n<|im_start|>user\noh i forgot to mention is uh for my wife not for me<|im_end|>\n<|im_start|>assistant\nno problem at all so we're looking for a 
medium-sized linen shirt for your wife are there specific colors or styles she prefers<|im_end|>\n<|im_start|>user\ncan we have a ping one<|im_end|>\n<|im_start|>assistant\na pink 
linen shirt sounds lovely would you like me to help find options for a medium-sized pink linen shirt<|im_end|>\n<|im_start|>user\nyeah medium size please", "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
                 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 0.11124372482299805, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:14:46.424 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "b98b13eabe58", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:48.547 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:53.530 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:57.392 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Have you done?", "language": "en", "transcript_delay": 0.5476160049438477, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:57.466 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.0007561199017800391, "duration": 0.06, "input": "<|im_start|>assistant\nno problem at all so we're looking for a 
medium-sized linen shirt for your wife are there specific colors or styles she prefers<|im_end|>\n<|im_start|>user\ncan we have a ping one<|im_end|>\n<|im_start|>assistant\na pink 
linen shirt sounds lovely would you like me to help find options for a medium-sized pink linen shirt<|im_end|>\n<|im_start|>user\nyeah medium size 
please<|im_end|>\n<|im_start|>assistant\ni will look for medium-sized pink linen shirts please hold on a moment<|im_end|>\n<|im_start|>user\nhave you done", "room": 
"console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:58.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:14:59.845 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 2.4524879455566406, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:15:00.122 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "90d58aa61035", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:03.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:08.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:13.542 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:16.678 DEBUG    livekit.agents     received user transcript  
                                         {"user_transcript": "Yes, please.", "language": "en", "transcript_delay": 0.5329608917236328, "room": "console-c59e1513", "pid": 86422, 
"job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:16.828 DEBUG    livekit.…_detector eou prediction  
                                           {"eou_probability": 0.7901280522346497, "duration": 0.13, "input": "<|im_start|>assistant\na pink linen shirt sounds lovely would you like
me to help find options for a medium-sized pink linen shirt<|im_end|>\n<|im_start|>user\nyeah medium size please<|im_end|>\n<|im_start|>assistant\ni will look for medium-sized pink 
linen shirts please hold on a moment<|im_end|>\n<|im_start|>user\nhave you done<|im_end|>\n<|im_start|>assistant\ni can't browse the internet in real-time but you can check popular 
online retailers like amazon nordstrom or local clothing stores for medium pink linen shirts would you like tips on what to look for when shopping<|im_end|>\n<|im_start|>user\nyes 
please", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:16.829 DEBUG    livekit.agents     using preemptive generation  
                                         {"preemptive_lead_time": 0.15001392364501953, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:15:17.103 INFO     livekit.…s.inworld Context created confirmed by server  
                                           {"context_id": "96004c124f86", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:18.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:23.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:28.525 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:33.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:38.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:43.528 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:48.520 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:53.546 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:15:58.525 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:16:03.527 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
^C    01:16:07.871 INFO     livekit.agents     shutting down worker {"id": "AW_CPqRVu3CTqGt"}
    01:16:07.875 DEBUG    livekit.agents     input stream detached  
                                         {"participant": "<local-participant>", "source": "SOURCE_MICROPHONE", "accepted_sources": ["SOURCE_MICROPHONE"], "room": "console-c59e1513",
"pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:16:09.211 INFO     root               ignoring text stream with topic 'lk.agent.request', no callback attached  
                               {"room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:16:09.969 DEBUG    livekit.agents     stream closed  
                                         {"participant": "<local-participant>", "source": "SOURCE_MICROPHONE", "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y",
"room_id": "RM_BAqYvEiXs8ji"}
    01:16:10.885 DEBUG    livekit.agents     session closed  
                                         {"reason": "user_initiated", "error": null, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": 
"RM_BAqYvEiXs8ji"}
    01:16:10.889 DEBUG    livekit.agents     shutting down job task  
                                         {"reason": "", "user_initiated": false, "room": "console-c59e1513", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:16:10.890 INFO     livekit.agents     process exiting {"reason": "", "pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:16:11.148 DEBUG    livekit.agents     http_session(): closing the httpclient ctx {"pid": 86422, "job_id": "AJ_q8yrqiCP4J8Y", "room_id": "RM_BAqYvEiXs8ji"}
    01:16:11.342 INFO     livekit.agents     process exiting {"reason": "", "pid": 86416, "inference": true}

Here’s my agent code:
agent.py (5.7 KB)

and uv lock (renamed extension because this chat disallow .lock file)
uv.lock.txt (579.4 KB)

I’m on a Mac. Server is on Ubuntu. Both got the same issue.

10

Just to confirm, @Quoc_Ngo and @ThanhLupin are from the same company, based on email addresses, so I assume this is the same issue :slight_smile:

This is a very simple agent, if you are able to reproduce your issue with this code, it points to an issue with infrastructure, not implementation

I still suspect the root cause is related to your hosting environment, can you deploy your agent to LiveKit cloud and retest? (link) You get 1 free cloud agent with the build plan, so you wouldn’t have to modify any of your production projects.

11

What might be the hosting environment configurations/defaults that cause that behavior?

It’s exactly at 8 minute mark of every call, so looks like there’s a timeout somewhere, either on Livekit host or on our host. It’s unlikely the host config/settings though, because:

  1. It happens all of sudden on AWS EC2 Ubuntu server since Friday last week, without any infrastructure update. We never seen anything like that before.
  2. My computer on Mac is completely isolated from our server environment, but could still reproduce the issue.
12

Sorry for the delayed response. This will address part of the issue for the media track:
https://github.com/livekit/agents/commit/4d6835ea1ac708e0cd1a1b8509dcd62d258126c1

Still looking into the other part of the issue you reported.

13

The fix from @CWilson is already shipped. Commit 4d6835ea (“fix: do not republish tracks on reconnect”, David Zhao, today 18:30 UTC) is in livekit-agents==1.5.9 on PyPI as of 19:53 UTC today.

That commit directly addresses the symptom @ThanhLupin opened with: track settings (including mute state) being lost after resume. The previous behavior republished tracks on reconnect, which is why your TTS publication state reset.

Immediate unblocker:

uv add "livekit-agents~=1.5.9"

Re-run @Quoc_Ngo’s reproducer; the track-state-after-resume half should be gone.

The underlying disconnect at the 8-minute mark is separately tracked per @CWilson’s “still looking into the other part” note. Don’t expect 1.5.9 to stop the reconnect from happening; expect it to stop the track-state damage when it does.

14

Thanks for the updates, @Muhammad_Usman_Bashir. Looking forward to the fix for the underlying disconnect issue.

15

I saw a fix for this go out this morning. It should no longer be an issue for you.

16

Thanks @CWilson. We no longer see errors or resume/reconnect at 8 minute in the last 12 hours.