Issues with LeaveRequest with reason = 8 (migration)

Client SDKs
,
1

Hello community!

We run group sessions on LiveKit Cloud, browser clients, rooms of 10 to 15+ people, and since around August 17 we have had a growing number of reports of calls dropping and taking a long time to come back. The ones we could trace all map to the client receiving a LeaveRequest with reason = 8 (MIGRATION), action = RESUME.

As far as we understand, MIGRATION is what the server sends when it moves a room off a node it considers unhealthy. What we cannot explain is what happens next, because the same migration resolves in two very different ways.

Usually the resume is quick:

+0s LeaveRequest reason=MIGRATION action=RESUME

+2s connectionState: signalReconnecting → connected

274 of the 370 migrations we could follow through to a reconnect look like that (p50 2s, p95 3s). Most of the rest look like this:

+0s LeaveRequest reason=MIGRATION action=RESUME

+16s LeaveRequest reason=CONNECTION_TIMEOUT action=RESUME

+20s connectionState: reconnecting → connected

Both kinds happen in the same room, from the same migration, at the same moment. In 31 room-level migrations we have one participant back in 3s or less and another taking 15s or more. Everyone gets the signal within a couple of seconds of each other (median spread 2s), and the slow ones are not the first or last to get it (first 12 times, middle 27, last 6). It does follow the participant, though. Of the 67 people who were migrated more than once, 42 were fast every time, 5 were slow every time (one of them 5 out of 5 at 20 to 23s, in a room where the others resumed in 2s) and 16 had both.

Many of our sessions occurs in the US Central servers but this also happen on other locations.

Do you also have experienced Migrations issues? Do your migrations introduce timeouts as well?

Versions: livekit-client@2.21.0, livekit-server-sdk@2.13.0 (only used to mint tokens).

2

Do you have a list of those “most of the rest” session IDs? (starts witih RM_) Or at least a subset of those sessions rather than all 96? I could look for patterns.

3

Hey @darryncampbell ,

Unfortunately, I wasnt able to export the LiveKit session IDs from our logs.

I’ve attached a CSV with the cases where the clients reconnected. For each migration, it includes:

  • the time between receiving the MIGRATION request and reaching connected again;
  • the timestamp when the migration request was received;
  • the livekit’s room UUID; and
  • the participant identity.

You’ll also find another file without the duration for clients that did not reconnected (abandoned the page or reloaded into a new session hence a different log trace).

Hopefully this is enough to help identify some patterns on your side. If there’s any other information we can extract that would help correlate these with LiveKit’s logs, please let me know.

results-never-reconnected.csv (5.1 KB)

results-reconnected.csv (38.4 KB)

4

For openness, I handed this over to Claude to analyse and look for patterns that could explain what you’re seeing.

The analysis came back quite sensible, with migrations being caused by normal load balancing on our Chicago node, where most of your rooms are created. Although your traffic didn’t change, that node started handling more traffic around the time you see the spike in migrations. Again, this is expected.

It looks like the issue is on the client-side, rather than on the server. There is no obvious correlation of why some clients experienced the issue, and others didn’t, but I do notice that updates were made to the reconnection logic in the latest 2.22.0, so I recommend updating to that: Release v2.22.0 · livekit/client-sdk-js · GitHub

It’s not a fix, but the delta between your 0s and +16s is being driven by peerConnectionTimeout, RoomConnectOptions | LiveKit JS Client SDK - v2.22.0 , (default 15s), so you should be able to lower that to reduce the 20s wait.

Sorry I don’t have a more black and white explanation for you.

5

Thanks, Darryn!

I’m Felipe’s co-worker. Indeed there is this PR / fix on latest 2.22.0 livekit version that aims to cover exactly the scenario where users receive the LeaveRequest 8 (MIGRATION) then have to wait for the LeaveRequest 14 (CONNECTION_TIMEOUT) to actually reconnect.

We have upgraded the client library to improve this, and we will monitor to see if now at least most of the reconnects happens in a few seconds.

One thing that I think won’t be solved is that whenever a MIGRATION happens, even when it reconnects in ~3s, it is too disruptive and makes users perceive our platform as unstable and low quality. A user reported that a migration happens to them twice a day on average.

I don’t have collected proper statistics and hard evidence yet, but it seems that migrations happen way more frequently for a set of users / livekit rooms. Here are my thoughts on this (didn’t collect hard evidence to confirm them):

  • Livekit rooms on US Central seems to have more migrations
  • Livekit rooms with more participants (between 12 and 20) seems to have more migrations
  • Livekit rooms that lasts longer (multiple hours) seems to have more migrations

We will keep monitoring now with the newest version of the livekit client, and will come back with more details if the migrations keep happening too often. In the meantime, do you have any insights regarding migrations happening too often?

Thank you so much for the help so far!

6

Hopefully the 2.22.0 resolves the connection timeout scenario.

Let me check on the infrastructure side if there is something unique to Chicago :eyes:

7

Regarding the timeout on reconnecting after a migration

Unfortunately the upgrade to 2.22.0 didn’t fix the (re)connection timeout scenario.

I followed the code path and the logs thoroughly for the most problematic user in a few occurrences, and I see that always after the MIGRATION (log line `client leave request received (action=1)` with reason 8), the js client logs the signal reconnecting (websocket) successfully and quickly, then the `triggering ICE restart` and nothing else happens until the timeout (log line `client leave request received (action=1)` with reason 14) or the user closes the page.

So, I wonder if there is a re-negotiation issue that gets in a deadlock for some users only when they restart ice, or only when a migration happens.

Regarding migrations happening

I noticed that on August 26th, about the same amount of livekit rooms were migrated for us

8

Are you able to share those client logs with me over DM? I’ll take a look

Still chasing the infrastructure angle, so there’s still two active questions:

  1. Are you seeing an unexpectedly high number of migrations
  2. When a migration does happen, why does it lead to connection timeouts.
9

Hey @darryncampbell,

Regarding your open questions, here is some data from our side to support the investigation:

  1. I can see a clear, sustained increase, roughly tripling over the month and peaking on Aug 24–25.

    I grouped the raw log rows into unique occurrences by clustering events that share the same room and fall within 120 seconds of each other. This translates the pattern where “everyone in the room gets the leave request at the same moment” into a single occurrence.

    That yields 224 unique migration occurrences affecting 488 participant-slots.

    DateOccurrencesParticipants affected
    2026-08-0333
    2026-08-041011
    2026-08-0522
    2026-08-06815
    2026-08-07411
    2026-08-10413
    2026-08-111325
    2026-08-121219
    2026-08-131226
    2026-08-1445
    2026-08-1511
    2026-08-17810
    2026-08-181115
    2026-08-191739
    2026-08-202156
    2026-08-211352
    2026-08-2223
    2026-08-242958
    2026-08-252669
    2026-08-261935
    2026-08-27520

    Full disclosure: I used Claude to access Sentry via MCP and to curate the logs.

    1. I’m going to DM you the log files for all participants who received a leave request with reason=8, followed by another with reason=14 (timeout).

    This is a subset of the numbers above because we had instances where people simply refreshed the page and were admitted back into the room.

    Thank you!

10

Hi @Felipe_Lube thanks for those logs but I’m sorry I wasn’t clearer. My reason for asking was to try and understand why your upgrade to 2.22.0 did not fix the (re)connection timeout scenario (i.e. it was taking a long time to reconnect after migration). Since you only upgraded to 2.22.0 on 27th, I was hoping you could share some logs from that point on, to help diagnose any client issues.

On the infrastructure side, and the question of ‘why are you seeing a high number of migrations in the first place’: I have been working with the infrastructure team since Thursday and, looking at the dashboard, the number of migrations in Chicago was substantially lower yesterday and Friday across all projects, so I would expect a corresponding reduction for yourself also.

11

Hello @darryncampbell !

I see. I’m going to sent you one instance we had yesterday.

About the reduction, that’s good news! On our side, it seems the number of leave signals of 8 followed by a 14 dropped to just one yesterday. However, we’re still seeing a high number of reason=14 disconnects (9 yesterday) and our peak usage historically has been on Tuesdays, so I’ll keep monitoring this through the coming week.

12

Thanks for that log you sent over DM. I took a look and there doesn’t seem to be anything wrong on the client side. Still investigating but:

that comment still applies when your clients are affected by this ^. Reducing this to something like 6-8 seconds would save you 7-9 seconds during a reconnection.

13

Hey @darryncampbell,

We discussed lowering this timeout, but the current value seems to fit our use case well, especially given the wide range of devices we support.

We’re also eager to see the effects of PR #2054, released in 2.22.2, since it seems to address the stall we were seeing after a migration. We updated our app today.

That said, the number of occurrences has already dropped dramatically. Yesterday, we didn’t observe any cases of reason 8 followed by reason 14. All four reconnections after a migration we had yesterday completed quickly, within 3 seconds.

The infrastructure changes you and the team made seem to be paying off. Thanks again!

14

Good spot - I had seen the investigation that led to #2054 but I hadn’t appreciated the similarity until I just read those patch release notes.

Really glad to hear this, thank you for your persistence with the issue.