DTLS timeout after ~10s with TURN/TCP in multi-node setup (v1.9.12)

Self Hosting
1

Hi everyone

We’re running a self-hosted multi-node LiveKit deployment across two regions (Germany on-prem + Qatar on GCP). Both nodes run on microk8s with hostNetwork: true. Everything works great within the same node, but we’re hitting a consistent DTLS timeout when participants connect cross-region via TURN/TCP.

Would really appreciate any insights from the community or the LiveKit team — especially from anyone running a similar on-prem multi-region setup with Kubernetes.

Would really appreciate any insights from the community or the LiveKit team — especially from anyone running a similar on-prem multi-region setup.

Setup

  • LiveKit v1.9.12, self-hosted, 2 nodes
  • Node 1: Germany (on-prem), Node 2: Qatar (GCP me-central1)
  • Shared Redis via WireGuard tunnel (~123ms latency)
  • TURN enabled with TLS passthrough via Contour/Envoy on port 443
  • Both nodes registered in Redis, signaling works correctly

Problem

When a participant connects from the Gulf region to a room hosted on the Germany node:

  1. WSS signaling connects fine (proxied via Qatar → Germany)
  2. ICE resolves to Germany TURN server via turns: on TCP 443
  3. Media works for ~10 seconds (audio + video both directions)
  4. Then: dtls timeout: read/write timeout: context deadline exceeded
  5. Video freezes, signaling stays connected

Same-node connections work perfectly. Issue only occurs in cross-node scenarios.

What we tried

  • packet_buffer_size_video: 5000, packet_buffer_size_audio: 2000
  • OS UDP buffers increased to 5MB (rmem_max, wmem_max)
  • Confirmed TCP 7881 connectivity between nodes
  • RTT ~130ms between client and TURN server

TURN Config

turn:
  enabled: true
  tls_port: 3478
  udp_port: 443

Questions

  1. Is there a configurable DTLS timeout or keepalive interval for high-latency TURN/TCP scenarios?

  2. We’re using Contour/Envoy as a reverse proxy for TLS termination (WSS) and TLS passthrough (TURN). Could this be causing the DTLS timeout? What reverse proxy setup do you recommend for self-hosted on-prem deployments?

  3. For those running multi-node LiveKit on-prem — what does your production setup look like in terms of reverse proxy, TURN, and TLS? Any gotchas with high-latency cross-region TURN/TCP?

  4. Any recommended configuration for self-hosted multi-node deployments with 100ms+ RTT between client and TURN?

2

Hey @hansama, did you manage to find a solution for this? Encountering a very similar situation

3

@hansama, @Philippe_Castonguay

Two diagnostic captures that would localize where DTLS is dying after the 10s mark.

  1. PCAP at both ends of the TURN/TCP relay (client edge and Germany node edge), filtered to TCP port 443. Look for whether DTLS records are still being relayed bidirectionally past the 10s mark, or whether the TCP connection itself is being torn down. That splits “DTLS server-side context dropped” from “TCP killed by something in the path.”

  2. Envoy access logs on the TURN/TLS-passthrough listener for a failing session. Specifically check stream_idle_timeout and idle_timeout closes. Envoy’s TLS-passthrough mode has idle-timeout knobs that are easy to leave at defaults set for short-lived HTTPS rather than long-lived RTC media. LK’s deployment docs cover TURN config (tls_port, udp_port, use_external_ip) [ Deploying LiveKit | LiveKit Documentation ] but don’t specifically address reverse-proxy idle timeout interaction with long-lived media.

If (1) shows DTLS still flowing but the LK server stops responding past 10s, that’s a server-side state issue and worth a livekit/livekit issue with the PCAP attached. If (2) shows Envoy closing on idle timeout, raise idle_timeout to several minutes on the TURN passthrough listener.

If both look clean, the next layer down is the WireGuard tunnel. MTU mismatch causes post-handshake media frames to fragment, and intermediate routers sometimes drop fragments after the low-rate DTLS handshake clears. Set MTU to 1380 or lower on the WireGuard interface and retest.

4

When a Gulf user joins a room hosted on the Germany node:

  1. Gulf user connects WSS to Qatar node (livekit-gulf.domain.com)
  2. Qatar node checks Redis, sees the room is on Germany node
  3. Qatar node acts as a signaling bridge — proxies WebSocket messages to Germany node via internal network
  4. For media (audio/video), the Gulf user’s browser gets ICE candidates pointing to Germany’s TURN server (livekitturn.domain.com:443)
  5. Browser establishes TURNS/TCP connection to Germany’s TURN server through HAProxy TLS passthrough
  6. Media flows via this TURN relay to the Germany node where the room lives

So the Qatar node only handles signaling proxy. Actual media goes directly from the Gulf client to the Germany TURN server via TURNS/TCP 443.

This is why HAProxy with long timeouts was critical — Envoy was killing the TURN/TCP connection after ~10 seconds

I switched from Envoy to HAProxy.

5

Glad you tracked it down. For future readers hitting the same symptom on Envoy without switching proxies, the equivalent fix is raising the TCP proxy idle_timeout on the TURN listener to several minutes (or 0 to disable).

The same logic applies to any reverse proxy default-tuned for HTTPS: long-lived RTC media looks idle to a proxy whose heuristics are calibrated for request-response traffic. HAProxy’s equivalents are timeout client and timeout server.