After `500 no media node available`, workers get no dispatch until manually restarted — why?

Getting Started
1

Severity: ~30-min agent outage; recovered only by manually restarting our worker pods.

Setup: livekit-agents 1.4.6 / livekit 1.1.2 / livekit-api 1.0.7 (Python 3.11), SIP telephony, self-hosted workers on Kubernetes, LiveKit Cloud project p_3tqm7ro6kbs, agent name prod_inbound.

Timeline (UTC, 2026-06-22)

Time What happened
12:00–12:21 Normal: ~15–23 job dispatches/min.
12:22–12:23 Dispatch burst, ~2× normal (37→43/min).
12:24:30–12:24:50 signal connection failed on v0 path: WsError(Http(status: 500, body: "no media node available")) — 56 occurrences across ~26 calls; all HTTP 500 no media node available, 0 timeouts.
12:25–12:29 Activity tapers across all worker pods.
~12:29:40 All worker pods go silent within ~3 seconds of each other (last log line on each). No crash, no exception, container RESTARTS=0.
12:30–12:45 Zero jobs dispatched to our workers. Workers emit essentially no logs (3–6 lines/min fleet-wide). LiveKit routes no traffic to us.
12:45:50 We manually drain/delete the worker pods (shutting down worker).
12:46:28–12:46:35 New pods start and registered worker with LiveKit.
12:46+ Traffic resumes immediately; backlog surge (49–63 dispatches/min) through 12:57.

Sample log lines (signal failures)

All 56 occurrences carry the same signature — an HTTP 500 whose body is Some([110, 111, 32, ...]), which decodes to no media node available. They span multiple workers (AW_JXrHXWM6qDiv, AW_dC4dDTfQFe4c, …) — i.e. fleet-wide, not one worker. Representative lines (phone numbers masked):

2026-06-22T12:24:36.573Z  worker=AW_JXrHXWM6qDiv  job=AJ_ZfnLVLYvJ3Q7  room=RM_5Jp3XZA2SBVV
  livekit_api::signal_client:287 - signal connection failed on v0 path:
  WsError(Http(Response { status: 500, version: HTTP/1.1,
    headers: {"content-length": "23", "content-type": "text/plain; charset=utf-8", ...},
    body: Some([110,111,32,109,101,100,105,97,32,110,111,100,101,32,97,118,97,105,108,97,98,108,101]) }))
  # body bytes decode to: "no media node available"

2026-06-22T12:24:36.651Z  worker=AW_JXrHXWM6qDiv  job=AJ_m9Kb5ymksaJq  room=RM_nfv87PjCV4mZ  (500, no media node available)
2026-06-22T12:24:37.006Z  worker=AW_JXrHXWM6qDiv  job=AJ_NC2kFQP4Q8yw  room=RM_gdV3vYQDApHk  (500, no media node available)
2026-06-22T12:24:37.612Z  worker=AW_dC4dDTfQFe4c  job=AJ_KrgCxarCuXAy  room=RM_e8pxiD2G6HJS  (500, no media node available)
2026-06-22T12:24:37.975Z  worker=AW_dC4dDTfQFe4c  job=AJ_KrgCxarCuXAy  room=RM_e8pxiD2G6HJS  (500, retry — same error)

What we observe

  1. The trigger is a server-side 500 no media node available storm — LiveKit had no media (SFU) node for a burst of connections at 12:24.
  2. Shortly after, our workers received no dispatched jobs at all for ~16–20 minutes, and our worker processes stopped logging (became unresponsive — no exception/crash, no container restart).
  3. Recovery required a manual worker restart. Dispatch did not resume on its own once media nodes were presumably available again; it resumed only after the old pods were deleted and new pods re-registered.

We have ruled out our post-call/egress pipeline as the cause: the post-processing service (lv-webhook) stayed healthy throughout and completed requests in seconds — it only went idle because the workers stopped sending it work.

Questions

  1. After returning 500 no media node available to a worker’s signal connects (a burst), does LiveKit de-register or mark that worker unavailable server-side? What was the registration/availability state of our workers from your side during 12:30–12:46 UTC?
  2. Why did job dispatch not resume on its own once capacity was available — i.e., why did our workers receive zero traffic until we manually restarted them?
  3. Is there a known failure mode where a 500 signal-connect storm leaves a self-hosted worker in a state where LiveKit stops dispatching to it until it re-registers (restart) — and conversely, can a 500 storm cause the agents worker to become unresponsive / drop its registration WebSocket?
  4. What is the recommended way to detect and auto-recover from this without a manual restart (worker health check, forced re-registration, reconnect/backoff config)? We’d like the system to self-heal rather than require human intervention during an outage.

Affected calls — rooms that hit no media node available (project p_3tqm7ro6kbs, 12:24:30–12:24:50 UTC)

RM_Jc4w4YFbhcY8   RM_TMmvm7in8uTH   RM_TbstwwhbXhoD   RM_y2sBNGKAFCbc
RM_X5QYwAH4zb2b   RM_zphvJX7NWj39   RM_RNoWVzntgK7q   RM_FEvMuFs6Af4T
RM_UjKZ5PYrcwKF   RM_d96hZW5Qxnoh   RM_XTEe7Q3KU4V9   RM_Hy5pRak4xGxr
RM_xzvcDjQoSbAx   RM_7KfacBkp8JUY   RM_5Jp3XZA2SBVV   RM_ZNp43Bvkoazk
RM_wEBKDxkt2gqg   RM_FbnChrSerH9K   RM_2u4QKoL2rxU2   RM_ws6DWFrVHypm
RM_nfv87PjCV4mZ   RM_7asB62UEfzhK   RM_gdV3vYQDApHk   RM_e8pxiD2G6HJS
RM_oHnNyBDWc7tR

Happy to share full worker logs, worker IDs, and job IDs for the window.

2

Thanks for the thorough report Zaheer, I have shared it with the team investigating the ongoing incident as it helps us understand the true impact: