# After \`500 no media node available\`, workers get no dispatch until manually restarted — why? **URL:** \ **Category:** Getting Started\ **Created:** [June 22, 2026, 1:33pm UTC](https://community.livekit.io/t/after-500-no-media-node-available-workers-get-no-dispatch-until-manually-restarted-why/1484 "2026-06-22T13:33:15Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Zaheer\_Abbas](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/zaheer_abbas/32/268_2.png) [@Zaheer\_Abbas](https://community.livekit.io/u/Zaheer_Abbas)\ **Post date:** [June 22, 2026, 1:33pm UTC](https://community.livekit.io/t/after-500-no-media-node-available-workers-get-no-dispatch-until-manually-restarted-why/1484/1 "2026-06-22T13:33:15Z")
**Severity:** ~30-min agent outage; recovered only by manually restarting our worker pods. **Setup:** `livekit-agents 1.4.6` / `livekit 1.1.2` / `livekit-api 1.0.7` (Python 3.11), SIP telephony, self-hosted workers on Kubernetes, LiveKit Cloud project `p_3tqm7ro6kbs`, agent name `prod_inbound`. ## Timeline (UTC, 2026-06-22) | Time | What happened | | --- | --- | | 12:00–12:21 | Normal: ~15–23 job dispatches/min. | | 12:22–12:23 | Dispatch burst, ~2× normal (37→43/min). | | **12:24:30–12:24:50** | **`signal connection failed on v0 path: WsError(Http(status: 500, body: "no media node available"))`** — 56 occurrences across ~26 calls; **all HTTP 500 `no media node available`** , 0 timeouts. | | 12:25–12:29 | Activity tapers across all worker pods. | | **~12:29:40** | All worker pods go **silent within ~3 seconds of each other** (last log line on each). No crash, no exception, container `RESTARTS=0`. | | **12:30–12:45** | **Zero jobs dispatched to our workers.** Workers emit essentially no logs (3–6 lines/min fleet-wide). LiveKit routes no traffic to us. | | **12:45:50** | We manually drain/delete the worker pods (`shutting down worker`). | | **12:46:28–12:46:35** | New pods start and **`registered worker`** with LiveKit. | | 12:46+ | Traffic resumes immediately; backlog surge (49–63 dispatches/min) through 12:57. | ## Sample log lines (signal failures) All 56 occurrences carry the same signature — an HTTP `500` whose body is `Some([110, 111, 32, ...])`, which decodes to **`no media node available`**. They span multiple workers (`AW_JXrHXWM6qDiv`, `AW_dC4dDTfQFe4c`, …) — i.e. fleet-wide, not one worker. Representative lines (phone numbers masked): ```auto 2026-06-22T12:24:36.573Z worker=AW_JXrHXWM6qDiv job=AJ_ZfnLVLYvJ3Q7 room=RM_5Jp3XZA2SBVV livekit_api::signal_client:287 - signal connection failed on v0 path: WsError(Http(Response { status: 500, version: HTTP/1.1, headers: {"content-length": "23", "content-type": "text/plain; charset=utf-8", ...}, body: Some([110,111,32,109,101,100,105,97,32,110,111,100,101,32,97,118,97,105,108,97,98,108,101]) })) # body bytes decode to: "no media node available" 2026-06-22T12:24:36.651Z worker=AW_JXrHXWM6qDiv job=AJ_m9Kb5ymksaJq room=RM_nfv87PjCV4mZ (500, no media node available) 2026-06-22T12:24:37.006Z worker=AW_JXrHXWM6qDiv job=AJ_NC2kFQP4Q8yw room=RM_gdV3vYQDApHk (500, no media node available) 2026-06-22T12:24:37.612Z worker=AW_dC4dDTfQFe4c job=AJ_KrgCxarCuXAy room=RM_e8pxiD2G6HJS (500, no media node available) 2026-06-22T12:24:37.975Z worker=AW_dC4dDTfQFe4c job=AJ_KrgCxarCuXAy room=RM_e8pxiD2G6HJS (500, retry — same error) ``` ## What we observe 1. The trigger is a server-side **`500 no media node available`** storm — LiveKit had no media (SFU) node for a burst of connections at 12:24. 2. Shortly after, **our workers received no dispatched jobs at all for ~16–20 minutes** , and our worker processes stopped logging (became unresponsive — no exception/crash, no container restart). 3. **Recovery required a manual worker restart.** Dispatch did **not** resume on its own once media nodes were presumably available again; it resumed only after the old pods were deleted and new pods re-registered. We have ruled out our post-call/egress pipeline as the cause: the post-processing service (lv-webhook) stayed healthy throughout and completed requests in seconds — it only went idle because the workers stopped sending it work. ## Questions 1. After returning `500 no media node available` to a worker’s signal connects (a burst), **does LiveKit de-register or mark that worker unavailable server-side?** What was the registration/availability state of our workers from your side during **12:30–12:46 UTC**? 2. **Why did job dispatch not resume on its own** once capacity was available — i.e., why did our workers receive zero traffic until we manually restarted them? 3. Is there a known failure mode where a `500` signal-connect storm leaves a self-hosted worker in a state where **LiveKit stops dispatching to it until it re-registers (restart)** — and conversely, can a `500` storm cause the agents worker to become unresponsive / drop its registration WebSocket? 4. What is the **recommended way to detect and auto-recover** from this without a manual restart (worker health check, forced re-registration, reconnect/backoff config)? We’d like the system to self-heal rather than require human intervention during an outage. ## Affected calls — rooms that hit `no media node available` (project `p_3tqm7ro6kbs`, 12:24:30–12:24:50 UTC) ```auto RM_Jc4w4YFbhcY8 RM_TMmvm7in8uTH RM_TbstwwhbXhoD RM_y2sBNGKAFCbc RM_X5QYwAH4zb2b RM_zphvJX7NWj39 RM_RNoWVzntgK7q RM_FEvMuFs6Af4T RM_UjKZ5PYrcwKF RM_d96hZW5Qxnoh RM_XTEe7Q3KU4V9 RM_Hy5pRak4xGxr RM_xzvcDjQoSbAx RM_7KfacBkp8JUY RM_5Jp3XZA2SBVV RM_ZNp43Bvkoazk RM_wEBKDxkt2gqg RM_FbnChrSerH9K RM_2u4QKoL2rxU2 RM_ws6DWFrVHypm RM_nfv87PjCV4mZ RM_7asB62UEfzhK RM_gdV3vYQDApHk RM_e8pxiD2G6HJS RM_oHnNyBDWc7tR ``` Happy to share full worker logs, worker IDs, and job IDs for the window. ---
**Author:** ![darryncampbell](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/darryncampbell/32/20_2.png) [@darryncampbell](https://community.livekit.io/u/darryncampbell)\ **Post date:** [June 22, 2026, 1:38pm UTC](https://community.livekit.io/t/after-500-no-media-node-available-workers-get-no-dispatch-until-manually-restarted-why/1484/2 "2026-06-22T13:38:21Z")
Thanks for the thorough report Zaheer, I have shared it with the team investigating the ongoing incident as it helps us understand the true impact: > **[Connection failures and SIP transfer errors in US Central](https://status.livekit.io/incidents/pg8h9qtm88mw)** > > LiveKit's Status Page - Connection failures and SIP transfer errors in US Central. ---
**Author:** ![system](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/system/32/26_2.png) [@system](https://community.livekit.io/u/system)\ **Post date:** [July 7, 2026, 1:38pm UTC](https://community.livekit.io/t/after-500-no-media-node-available-workers-get-no-dispatch-until-manually-restarted-why/1484/3 "2026-07-07T13:38:43Z")
This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.