Load test at 1,000 participants: ~19% of subscriber connections fail with "could not establish signal connection" — CPU/RAM not saturated

Self Hosting
,
1

Hello LiveKit community,

I’m self-hosting LiveKit via Docker Compose and load-testing with the lk CLI. At 1,000 simulated participants, 189 of 975 subscriber connections (~19%) failed with:

could not connect Sub 808: could not establish signal connection

Around the same timestamps, the TURN client logs show:

turnc ERROR: Fail to refresh permissions: all retransmissions failed for <participant>
turnc ERROR: Fail to refresh permissions: transaction closed

Failures are concentrated among the later-joining participants (roughly sub #794–999, the tail of the ramp-up), not spread evenly across all 1,000 consistent with a connection-burst/timing issue rather than a hard capacity ceiling.

Server resources were not saturated during the failure window — docker stats sampled every 2s for the whole run shows the LiveKit container peaking at 482.93% CPU (of 800% / 8 cores, ~60%) and 2.8 GiB / 7.75 GiB RAM. Redis and the API container also stayed well under load in comparable runs. This is why I don’t think this is a raw CPU/RAM/FD-limit problem, but I’d like the community’s read on it.

Environment

  • Deployment: Self-hosted, Docker Compose, single host
  • LiveKit server: livekit/livekit-server:v1.11.0
  • LiveKit CLI (lk): v2.16.7
  • Server: 8 cores / 7.8 GB RAM
  • Redis: 7.4.8 (single instance, redis:7-alpine)
  • Reverse proxy: Nginx (LiveKit itself runs with network_mode: host, not behind Docker bridge NAT)
  • TURN/TLS: enabled (domain, tls_port: 5349, cert configured in livekit.yaml)
  • Load-test command:
lk load-test \
  --url wss://<my-livekit-domain> \
  --api-key $LIVEKIT_API_KEY \
  --api-secret $LIVEKIT_API_SECRET \
  --room stepped-loadtest-1000 \
  --audio-publishers 5 \
  --subscribers 995 \
  --num-per-second 10 \
  --duration 160s

(ramp rate: 10 participants/sec, so ~1000 participants joining over ~100s)

Known confound: the load-test client runs on the same 8 cores as the server under test, so part of the load at the moment of failure is the test tool’s own CPU cost (invisible to docker stats), not purely server-side. A second-machine retest is on my list but hasn’t happened yet.

Expected behavior

All 1,000 simulated participants connect, or the failures scale roughly proportionally with load rather than clustering sharply among the last-joining participants.

Actual behavior

189/975 subscribers failed with could not establish signal connection, clustered in the tail of the ramp, alongside TURN “fail to refresh permissions” errors on the same connections around the same timestamps.

Questions

  1. Is “Fail to refresh permissions: transaction closed / all retransmissions failed” a downstream symptom of the signaling/ICE connection already failing or timing out, or could TURN allocation itself be the root cause of the signal-connection failure? (My working assumption is the former the transaction closes because the underlying connection attempt failed, but I’d like to confirm.)
  2. Does Sub 808 etc. correspond directly to the Nth simulated subscriber in lk load-test’s connection order?
  3. Given CPU/RAM headroom, is there a known per-node concurrency limit on simultaneous ICE/DTLS handshakes (vs. steady-state media forwarding) that a connection burst could hit even while average resource usage looks fine?
  4. Is --num-per-second 10 too aggressive for a 1,000-participant single-room ramp? Is there a recommended ramp rate/burst size for large-scale lk load-test runs?
  5. Are there Prometheus metrics or debug logs specifically for ICE/TURN handshake latency or signal-connection setup time (as opposed to steady-state track stats) that would help pinpoint where in the handshake this is timing out?

Thank you.

2

@Mohammad_Khdour, Your co-located generator is the prime suspect, and docker stats cannot see it. The tester opens a real WebRTC peer connection per subscriber, and the benchmark guide says to run it on a separate machine with plenty of CPU, bandwidth, and file handles (benchmark). Running 1000 of those on the same 8 cores as the server means client and server fight for CPU at the ramp peak, while docker stats only measures the server container (~60%). Failures clustering in the tail is the client-starvation signature, not a server ceiling.

Quick answers:

  • The turnc “all retransmissions failed” lines are your load-test client’s own TURN client timing out under that starvation, correlated with the signal failures, not a separate server TURN fault.
  • Sub 808 is the 808th subscriber launched: the tester names participants <prefix>_<sequence> in launch order (loadtester.go#L105), so #794-999 is just the tail of the ramp.
  • --num-per-second 10 is already the tool’s hard cap: it defaults to 5 and maxes at 10 (loadtest.go#L62). There is no fixed handshake-concurrency limit to hit; the cost is the concurrent ICE/DTLS/TURN setup itself, which on a shared host lands hardest on the client.
  • Hold off on metrics until the generator is off the box, since right now it is largely measuring itself.

Rerun clean, on a separate host with a gentler ramp:

lk load-test --url wss://<domain> --room stepped-loadtest-1000 \
  --audio-publishers 5 --subscribers 995 --num-per-second 5 --duration 160s

If the failures shrink or move, it was the ramp and host, not steady-state capacity..

3

Thanks for the detailed breakdown; really helpful.

Makes sense that the co-located generator was skewing the results, and that docker stats wouldn’t catch it since it only watches the LiveKit container.

I’ll rerun with your command from a separate machine and report back with the results.

Thanks again for the help.