Co-located egress uses the TURN relay instead of the SFU's advertised loopback candidate → corrupt HLS

Self Hosting
,
1

Co-located egress falls back to the TURN relay instead of a loopback path → corrupt HLS

Hi everyone

I run whentheycry.social, a small community social networking site. It’s a passion project: a library, forums, chat, and recently I added live streaming so members can broadcast to each other. There are two playback modes, a low-latency WebRTC mode and a buffered HLS “Smooth” mode built on LiveKit Egress.

LiveKit has been brilliant for everything else, the WebRTC side is rock solid. I’m stuck on one specific networking problem with the HLS egress and I’ve run out of ideas, so I’d really appreciate a steer from anyone who knows the egress/ICE internals. I’ve tried to do my homework below and include the exact configs and logs.

Thanks in advance

TL;DR

One box, everything host-networked, one public IPv4 and no NAT. A co-located egress subscribing to a track connects to the TURN relay on the box’s own public IP instead of going direct over loopback. That relay drops packets and the egress HLS output (SegmentedFileOutput) is visibly corrupt with choppy audio. A normal WebRTC subscriber to the exact same track is clean, so ingest and the SFU are fine, only the SFU → egress subscribe path is bad.

I already tried the obvious SFU-side fix, making the server advertise 127.0.0.1 as well as the public IP (verified in the server log). The egress still relays. egress.yaml has no ICE/rtc config at all.

Question: how do I make a same-host egress connect to the SFU over loopback (or pair with the advertised 127.0.0.1 host candidate) instead of falling back to the relay? Is there an egress-side equivalent of enable_loopback_candidate / an rtc block for egress.yaml?

Environment

  • Single bare-metal host (Hetzner), one direct public IPv4 88.198.26.24, no NAT in front. IPv6 is also present on the box.
  • livekit-server, livekit/ingress, livekit/egress all run network_mode: host (WHIP media requires it) and reach each other on 127.0.0.1.
  • Versions: livekit/livekit-server latest, livekit/egress:latest, livekit/ingress:latest. Egress uses server-sdk-go/v2, pion/ice/v4.
  • Path: remote OBS → WHIP → ingress (transcoding bypassed, H.264) → SFU → { WebRTC viewers | egress → SegmentedFileOutput → HLS }.

Topology

                       remote OBS (public internet)
                              │  WHIP / H.264
                              ▼
   ┌────────────────────────────────────── one host, all network_mode: host ────────┐
   │  ingress  ──ws://127.0.0.1:7880──►  livekit-server (SFU)                         │
   │                                        │                                         │
   │                          ┌─────────────┴───────────────┐                         │
   │            WebRTC (clean) ▼                             ▼ subscribe (CORRUPT)     │
   │                    remote viewers                     egress  ── HLS .ts/.m3u8   │
   └─────────────────────────────────────────────────────────────────────────────────┘

The egress and the SFU are the same machine on loopback, so a 127.0.0.1 ↔ 127.0.0.1 pair should be a guaranteed direct path. It is not being used.

Symptom

  • HLS output (egress SegmentedFileOutput) is corrupt: macroblocking, garbled/frozen frames, audio breaks up.
  • A WebRTC subscriber to the same track at the same time is perfect, so ingest and the SFU are healthy; the loss is on the SFU → egress subscribe connection.
  • The egress log shows it sitting on a relay candidate against the box’s public IP and timing out, never establishing a direct pair (see Evidence 2).

Configs (live, on the box, keys redacted)

livekit.yaml (SFU) - currently deployed (baseline)

port: 7880
prometheus:
  port: 6789
bind_addresses:
  - ""

rtc:
  tcp_port: 7881
  udp_port: 7882
  use_external_ip: false # tried `true`
 # advertise_internal_ip: true
 # enable_loopback_candidate: true
  node_ip: 88.198.26.24
  ips:
    excludes:
      - "::/0"
      - "172.16.0.0/12"

keys:
  <api_key>: <api_secret>

webhook:
  api_key: <api_key>
  urls:
    - http://localhost:2312/api/v1/livekit/webhook

turn:
  enabled: true
  udp_port: 3478

redis:
  address: 127.0.0.1:6380

ingress:
  whip_base_url: "https://ingress.whentheycry.social/w"

room:
 enabled_codecs:
  - mime: audio/opus
  - mime: video/H264
  - mime: video/VP8
  - mime: video/VP9
  - mime: video/AV1
  - mime: video/rtx

egress.yaml - currently deployed (note: no rtc / ICE block exists)

log_level: info

api_key: <api_key>
api_secret: <api_secret>

ws_url: ws://127.0.0.1:7880

redis:
  address: 127.0.0.1:6380

ingress.yaml - currently deployed (shown because the working WHIP path uses the same IP settings)

logging:
  level: info

api_key: <api_key>
api_secret: <api_secret>

ws_url: ws://127.0.0.1:7880

redis:
  address: 127.0.0.1:6380

whip_port: 8090
rtc_config:
  udp_port: 7885
  use_external_ip: false
  node_ip: 88.198.26.24
  ips:
    excludes:
      - "::/0"
      - "172.16.0.0/12"

cpu_cost:
  whip_bypass_transcoding_cpu_cost: 0.1
  whip_cpu_cost: 2
  rtmp_cpu_cost: 2
  url_cpu_cost: 2

What I tried on the SFU, and the proof it took effect

I switched the SFU rtc block to advertise the internal/loopback candidate alongside the public IP:

rtc:
  tcp_port: 7881
  udp_port: 7882
  use_external_ip: true
  advertise_internal_ip: true
  enable_loopback_candidate: true
  # node_ip removed (single-candidate replace mode was suppressing the loopback candidate)
  ips:
    excludes:
      - "::/0"
      - "172.16.0.0/12"

The server confirms it then advertises both 88.198.26.24 and 127.0.0.1:

INFO  livekit  rtcconfig/webrtc_config.go:105  using external IPs
  {"ips": ["88.198.26.24/88.198.26.24", "127.0.0.1/127.0.0.1"], "advertiseInternalIP": true}
INFO  livekit  rtcconfig/ip.go:211             found external IP via STUN
  {"localAddr": "88.198.26.24:7882", "stunServer": "global.stun.twilio.com:3478",
   "externalIP": "88.198.26.24", "validateExternalIP": true}

With the SFU offering 127.0.0.1 to subscribers, the egress still selected the relay and the HLS was still corrupt. Because that variant changed nothing for the egress (and use_external_ip: false is the only setting that keeps RTC + WHIP healthy on this box’s IPv6 setup), I reverted to the baseline above.

Evidence

1. SFU advertises the loopback candidate (shown above) - egress ignores it.

2. Egress is parked on the relay candidate and times out

WARN  egress.lksdk.pion.ice  Failed to read from candidate udp4 relay 88.198.26.24:NNNNN
  related 0.0.0.0:NNNNN: read udp 88.198.26.24:NNNNN: i/o timeout
WARN  egress.lksdk.pion.ice  Failed to ping without candidate pairs. Connection is not possible yet.

(Port numbers vary per session; refresh with docker logs <egress-container> 2>&1 | grep -iE "ice|relay|candidate" | tail -40.)

The egress never forms a direct candidate pair, so it lands on the relay, which on this single-public-IP box means hairpinning the box’s own public IP, that path is lossy, hence the corruption.

Already ruled out

  • SFU not advertising loopback - ruled out. With use_external_ip: true + advertise_internal_ip: true the SFU advertises 127.0.0.1 (log above) and the egress still relays.
  • node_ip replace mode hiding the loopback - addressed (removed node_ip in the variant above so 127.0.0.1 reaches the wire), no change to the egress.
  • Reverse-path filtering - loosened (net.ipv4.conf.all.rp_filter = 2). No change; the box still cannot hairpin its own public IP (no NAT reflection), which is why the public-IP host pair fails and ICE falls onto the relay.
  • Codec / B-frames - not it. Egress EncodingOptions is H264_BASELINE, KeyFrameInterval: 2, AAC; the WebRTC view of the same track is clean.
  • egress.yaml ICE config - there is no rtc block in egress.yaml, so I cannot see a documented way to make the egress gather its own loopback host candidate or prefer a local path.

Questions

  1. With the SFU advertising 127.0.0.1 to subscribers, why does a co-located egress not form a 127.0.0.1 ↔ 127.0.0.1 pair and instead fall back to the relay? Does the egress need to gather its own loopback host candidate for that pair to exist?
  2. Is there an egress-side equivalent of enable_loopback_candidate, or an rtc / ICE config block for egress.yaml? If so, what are the exact keys?
  3. What is the recommended way to run an egress on the same host as the SFU so its subscribe connection stays on loopback and never relays (force-local ICE, a dedicated internal interface, ICE-lite, etc.)?

Thank you so much!

Victoria M

2

Welcome, @Victoria. This is a real gap, not a misconfig. A 127.0.0.1 pair needs a loopback candidate on both ends; enable_loopback_candidate only covers the SFU side, and the egress (pion via server-sdk-go) gathers no loopback candidate, so there’s nothing to pair with and ICE drops to the relay, which here is your own TURN [ livekit/server-sdk-go engine.go ]. There’s no egress-side equivalent: the egress config has no rtc/ICE block at all [ livekit/egress base.go ].

Fix without touching use_external_ip (so your WHIP/IPv6 path stays put): give the box a private host IP and advertise it alongside the public one.

ip link add lk0 type dummy && ip addr add 10.0.0.1/24 dev lk0 && ip link set lk0 up


rtc:
  use_external_ip: false
  # drop node_ip: it forces single-IP "replace" mode, which suppressed your loopback
  ips:
    includes:            # CIDR allowlist, advertised as host candidates
      - 88.198.26.24/32  # public, for remote viewers
      - 10.0.0.1/32      # private, the egress pairs here

The host-networked egress gathers 10.0.0.1 on its own, so a direct pair forms instead of the relay [ livekit/mediatransportutil rtcconfig/webrtc_config.go ]. ips is global, so confirm remote viewers still connect after.

3

Thank you, that was the missing piece. With node_ip dropped, ips.includes advertising 10.0.0.1 alongside the public IP, and a dummy lk0 interface, the egress now gathers the 10.0.0.1 host candidate and its subscriber connection comes up connectionType: "udp" in ~16 ms. The TURN relay fallback is gone entirely. The relay problem is fully solved, thank you.

But the relay was never what was corrupting the recording, and to be clear, this is not stutter or dropped frames. On a clean direct UDP path the HLS output is completely destroyed: every single frame is a dense field of rainbow macroblock noise over a barely-recognisable image, the audio is shredded, the recording is totally unusable. Screenshot attached.

It looks like a NACK/retransmit problem, not a network one. The egress floods:

egress.lksdk.pion.srtp  srtp ssrc=… index=NNNNN: duplicated packet

thousands per second. My read: on this co-located path packets occasionally arrive reordered, the egress reads the gaps as loss and NACKs hard, the SFU retransmits via RTX, the originals (only late) then also arrive, and SRTP replay-protection discards the duplicates. So the retransmit machinery is manufacturing massive packet loss on a path that had essentially none, and it scales directly with bitrate.

Ruled out:

  • Not the relay - fixed, direct UDP.
  • Not CPU - egress sits at ~2.2 cores of 16 at 1080p60, ~0.4 cores at 720p30.
  • Not socket buffers - net.core.rmem_max/wmem_max both 16 MB.
  • Not the player or B-frames - baseline H.264, and a direct WebRTC subscriber to the same track is perfectly clean. Only the egress’s copy is destroyed.

At 720p30 the picture is clean but it constantly rebuffers, so the storm is just under the threshold there rather than gone.

The (new?) question: the egress is co-located with the SFU on a path with effectively zero real loss, so the NACK/RTX recovery is not just unnecessary, it’s the thing destroying the output. Is there any way to (a) make the egress subscriber tolerate reordering (a larger jitter/reorder buffer so it stops false-NACKing), or (b) reduce or disable NACK/RTX for a low-loss co-located subscriber? egress.yaml has no rtc/transport block, so I can’t see a knob for it.

Versions: livekit-server 1.13.1, egress latest (server-sdk-go v2.16.1, pion/ice v4.2.0).

As reference here is the image of the stream

2026-06-16_11-08-45
2026-06-16_11-08-451920×1080 1 MB

and this would basically run at about 1 frame per 5 seconds (ish), dropping to 720p30 fixes this issue almost entirely

4

@Victoria, Glad the loopback pair came up clean. The knob you thought didn’t exist does, just not as an rtc block; it’s the latency block, absent from your egress.yaml so you’re on defaults [ livekit/egress pkg/config/base.go ]:

latency:
  jitter_buffer_latency: 200ms   # reorder tolerance for sdk egress
  old_packet_threshold: 0        # 0 = keep late retransmits instead of dropping them

jitter_buffer_latency (“jitter buffer max latency for sdk egress”) is exactly the reorder buffer from your (a): raise it and the depacketizer waits out reordered packets instead of reading the gaps as loss and false-NACKing. old_packet_threshold controls whether the synchronizer discards the late RTX packets that land after the originals, the second half of your duplicate theory; 0 disables those drops.

That block is flagged in source as “modifying these may break the service,” so bump jitter_buffer_latency modestly (a few hundred ms) and watch the srtp duplicate rate rather than going large at once. There’s no NACK/RTX disable in egress config (only this latency block), and you shouldn’t need one if the buffer absorbs the reorder. If a verified zero-loss 1080p60 path still storms after tuning these, that’s an egress receive-pipeline bug worth a minimal-repro issue; I didn’t find an existing one.

5

Hi @Victoria

I am not an expert on egress/ingress or self-hosting so maybe @Milos_Pesic will correct me if I am wrong here.

Since this is a loopback interface with no loss, I suspect the cause is an unpaced burst overrunning the egress UDP socket buffer.

You can check this with a command like this:

netstat -su | grep -iE 'receive (buffer )?errors|RcvbufErrors'   # or: ss -uem

If RcvbufErrors climbs in lockstep with the corruption, it’s socket-buffer overflow, not the network. Then, roughly in order of effort:

  1. Raise the actual receive buffer, not just the max. rmem_default is what an un-tuned socket gets: sysctl -w net.core.rmem_default=8388608 (and keep rmem_max ≥ that). This is the most likely single fix and the cheapest to test.
  2. Shrink the burst at the source. OBS keyframe interval and bitrate drive keyframe size. A shorter GOP / lower bitrate (or simply not bypassing transcoding for the HLS path) keeps each burst under the buffer — which is why 720p30 already works.
  3. Move egress off the loopback path. Counterintuitively, a real low-loss NIC link paces better than instantaneous loopback delivery; co-locating maximizes burstiness. If it must stay co-located, the buffer bump in (1) is the lever.