Seeking strategies to optimize connection reliability and audio quality for users in strict GCC network environments experiencing aggressive UDP throttling

Hello everyone,

We are running a production real-time 1-on-1 tutoring web application built on LiveKit Cloud (React frontend + Node.js backend). While our global users experience flawless connectivity, we are facing significant audio degradation and connection issues specifically with users located in strict GCC regions (Saudi Arabia, Oman, and the UAE).

From analyzing our client-side WebRTC stats and raw telemetry, we have noticed two prominent behaviors indicating heavy network interference/VoIP firewalls:

  1. High Packet Loss on Direct Paths: Students using standard ISP configurations frequently show extremely high UDP packet loss (sometimes spikes of 40% to 80% on host/prflx candidates), alongside extreme round-trip times (RTT maxing out into multiple seconds).
  2. Forced TCP Fallback: In many instances, UDP ports are completely blocked or throttled via Deep Packet Inspection (DPI) by regional ISPs, forcing our application to rely heavily on TURN/TLS over TCP Port 443 via *.turn.livekit.cloud.

While the SDK successfully falls back to TCP to rescue the call, the inherent physics of TCP (Head-of-Line blocking) means late audio packets get thrown away by the browser’s jitter buffer, resulting in choppy audio and severe software echo cancellation failures.

Our Questions for the LiveKit Team:

  1. Leveraging the me (Middle East) Region: We saw in the documentation that LiveKit Cloud has a Middle East (me) region with nodes in Jeddah and Dubai. Since we don’t have the enterprise Scale plan for protocol-based region pinning yet, can we manually update our client connection strings to <subdomain>.me.rtc.livekit.cloud for GCC users to ensure they hit the local edge loop directly? Will this minimize cross-border DPI inspection?
  2. TURN Obfuscation / Optimization: When a GCC network blocks standard UDP ports (50000-60000) and pushes the user to a TURN relay, what are the best practices for optimizing the TURN layer? For instance, does LiveKit Cloud natively run strategies to mask WebRTC traffic over Port 443 to resemble standard HTTPS traffic to bypass state-level deep packet inspection?
  3. SDK Congestion Settings: Are there client-side configurations we should adjust specifically for high-loss/forced-TCP environments to prevent the browser’s engine from constantly dropping audio tracks when a network spike occurs?

Any insights from the team or others running apps in firewalled regions would be incredibly helpful!

Interesting, I haven’t seen similar reports from our other customers in the Middle East. If you can share a couple of sessions where you experience particularly poor audio and connection issues I can see if I can validate what you are seeing from our end.

[I removed the rest of my answer in lieu of the below]

Hey Faraan,

Darryns answer is spot on. I accidently duplicated effort here and had also looked at this, so will share my perspective on it as well since I already looked at it. I hope one of our answers will help you.

Also, thanks for the detailed writeup, the telemetry you’ve collected (loss spikes on host/prflx, multi-second RTT, forced TURN/TLS fallback) paints a very recognizable picture of GCC access networks. Good news up front: you’re closer to the right setup than you think, and the biggest lever available to you doesn’t require the Scale plan. Taking your three questions in order.

1. Do you need to manually target the me region?

Short answer: no — and manually rewriting the URL isn’t the supported way to do it anyway.

LiveKit Cloud routes every connection to the nearest healthy edge automatically, on every plan. When the SDK connects, it first hits a region-discovery endpoint that returns the closest regions to the client’s IP, sorted by proximity, and connects to the best one — falling back to the next-closest if that connection fails. Your Saudi and UAE students should already be landing on the me edges (Jeddah/Dubai) today without you doing anything.

You can verify this directly instead of guessing — after connect, log:

room.once(RoomEvent.Connected, () => {
  console.log('edge region:', room.serverInfo?.region);
});

Add that to the telemetry you’re already collecting. If GCC users are consistently landing somewhere other than me, that’s genuinely unexpected and worth a thread of its own with a few example session IDs.

Region pinning (the Scale-plan feature) solves the opposite problem: it restricts which regions may serve your project, for compliance/data-residency, and in doing so turns off failover to other regions. It doesn’t make routing “more local” than the default — the default is already nearest-edge.

One important caveat on expectations: connecting to a local edge shortens the network path and helps RTT, but the packet mangling you’re measuring on host/prflx candidates is happening in the student’s access network — the ISP’s DPI is throttling UDP before it ever leaves the country. A Jeddah edge doesn’t exempt that traffic from inspection. So this reduces cross-border latency, but it won’t fix the UDP story on its own. That’s what #2 is for.

2. TURN best practices for DPI-heavy networks

What LiveKit Cloud already does: TURN/TLS on *.turn.livekit.cloud is served on TCP 443 with a real TLS handshake and a valid certificate. To port-based and protocol-based DPI, that connection is indistinguishable from ordinary HTTPS — there’s no separate “masking” toggle because real TLS on 443 is the masking. (Like any HTTPS connection, the SNI is visible, so an ISP that block-lists *.turn.livekit.cloud by hostname could still interfere — but that’s not a pattern we’ve seen deployed broadly in GCC networks.)

The actual best practice for your situation is to stop letting badly-throttled users attempt UDP at all. When you can identify a user as being on a hostile network (geo-IP, ASN, or a failed prior session), force relay from the start:

await room.connect(url, token, {
  rtcConfig: { iceTransportPolicy: 'relay' },
});

This does three things for you:

  • Faster, more reliable setup — ICE doesn’t spend time probing host/srflx paths that show 40–80% loss.
  • No mid-call “UDP whack” — the classic DPI pattern is a UDP flow that works for 30–90 seconds until it’s classified, then gets throttled to death, forcing a disruptive mid-call migration. Starting on relay avoids ever being on that path.
  • Deterministic behavior you can reason about in telemetry, instead of a different ICE outcome per student per session.

Note that 'relay' still allows TURN/UDP where it works — the browser will pick the best relay transport, and falls back to TURN/TLS 443 when UDP is blocked outright. You don’t need (and can’t currently force) TLS-only.

For triage, have affected students run the LiveKit connection test — it exercises each transport in order (WebSocket, WebRTC/UDP, TURN, TCP-vs-UDP, closest region) and tells you exactly which rungs of the ladder their network allows. The same checks are available programmatically via the ConnectionCheck class in livekit-client, so you can build a pre-call network check right into your tutoring app:

flowchart TD
    A[ICE / UDP 50000-60000] -->|blocked or throttled| B[TURN/UDP]
    B -->|UDP blocked entirely| C[TURN/TLS on TCP 443<br/>looks like HTTPS to DPI]
    A -->|clean path| D[Direct media âś…]
    B -->|works| D
    C --> D

3. Client-side settings for high-loss / forced-TCP calls

A few things, roughly in order of impact for a 1:1 tutoring app:

Keep RED on (it’s on by default). livekit-client enables redundant audio encoding (RED) by default for mono Opus tracks — each packet carries a copy of earlier audio, so isolated UDP loss is nearly free to recover from. Just make sure nothing in your publish options sets red: false. Same for dtx, also on by default.

On forced-TCP paths, the enemy is queue depth, not loss. TCP retransmits everything, so “loss” surfaces as latency spikes that overflow the browser’s jitter buffer — which you can’t tune from the SDK. What you can control is how full the pipe is: a TCP connection running near its bandwidth limit turns every loss event into a multi-hundred-ms stall for the audio riding behind it. Concretely:

  • Cap video aggressively for relay/TCP users — for 1:1 tutoring you don’t need much. Set a conservative videoEncoding.maxBitrate, or drop to audio-only when quality degrades:
room.on(RoomEvent.ConnectionQualityChanged, (quality, participant) => {
  if (participant.isLocal && quality === ConnectionQuality.Poor) {
    room.localParticipant.setCameraEnabled(false); // protect the audio
  }
});
  • A lower audioPreset (e.g. AudioPresets.speech instead of the default) buys headroom on constrained links with little perceptual cost for voice.
  • Keep adaptiveStream: true so the subscribe side downshifts automatically.

On the echo cancellation failures: AEC breaking is usually downstream of choppy far-end playback — the canceller can’t model a reference signal full of gaps. In our experience you don’t fix AEC directly; you fix delivery (everything above) and AEC recovers with it.

If you instrument room.serverInfo?.region plus per-session iceTransportPolicy and connection-quality events, you’ll very quickly see which combination (local edge + forced relay + capped video) moves your GCC metrics. Happy to dig into specific session traces if you get stuck — a couple of room names/SIDs from bad sessions would be enough.

Some Sresource: Region pinning docs, Regions overview, Firewall configuration, Firewall troubleshooting KB, Connection test KB, Codecs & RED docs,

This is an incredibly helpful breakdown. Thank you for clarifying the Anycast routing and the “UDP Whack” pattern.
We have analyzed our telemetry and our implementation. We have some critical questions about the deployment strategy for our GCC students (Saudi Arabia/UAE/Oman) and the value proposition of a Native App versus our current Web implementation.

Our worst-performing students in Saudi Arabia and Oman perfectly align with a DPI firewall allowing UDP for a few seconds before aggressively throttling it, causing massive TCP queue buildup.

  • Student 1 (SA): 8 sessions. 41.72% of samples marked poor. Max RTT spiked to 10,808ms. Max Jitter 315ms.

  • Student 2 (OM): 6 sessions. Max RTT hit 10,303ms. Worst fraction lost: 91.4%.

Our Current React Implementation: Right now, our system relies on a custom Watchdog that reacts to these stalls. We explicitly set dtx: false so that a flat packet counter definitively means transport death. When we detect a 9-second flatline, we bump the room key and force a reconnect with iceTransportPolicy: "relay".

JavaScript

// publishDefaults
dtx: false,
audioPreset: { maxBitrate: 48_000 }, // Teacher is at 64_000
simulcast: true,

// Watchdog Recovery
void fetchRoomToken(roomName, displayName, `media-stall-${direction}`)
  .then((data) => {
    setConnectionDetails((prev) =>
      prev ? { ...data, key: prev.key + 1, forceRelay: true } : null
    );
  });

Question 1: Based on your previous advice, should we abandon this reactive watchdog for GCC users and just pass iceTransportPolicy: "relay" on initial mount based on their Geo-IP? Question 2: We are manually setting bitrates (48k/64k) instead of AudioPresets.speech. Is this contributing to the massive 10-second RTT queue buildups when we fall back to TCP?

Secondly,while the GCC data makes sense, we have a student connecting from the US whose telemetry totally contradicts the user experience.

  • US Student: 2 sessions. 0.00% poor samples out of 11,652 total samples.

  • Stats: Max RTT was only 134ms. Max Jitter was 127ms. Worst fraction lost was 5.1%.

On paper, this is a flawless connection. However, the student reported the audio was “very laggy.”

Our Mic Constraints are set as follows (relying on browser AEC):

JavaScript

{
  noiseSuppression: !willUseKrisp, 
  echoCancellation: true,           
  autoGainControl:  true,           
}

Question 3: When WebRTC stats show a perfectly healthy transport, but the user experiences heavy “lag” (delay), what is the typical culprit? Does the LiveKit client telemetry capture device-level latency (e.g., standard Bluetooth headphones adding 200ms of delay, or the browser’s AEC struggling on a slow CPU)?

And lastly- We are under pressure to reach literally “0 glitches.” We are considering wrapping our React app in a Native (React Native + LiveKit SDK) container.

  • The Question: Will a Native App implementation inherently provide better connectivity/firewall-bypassing capabilities than the Web SDK? Or are we just trading browser-sandbox limitations for development complexity without addressing the underlying ISP/DPI firewall throttling?

Current Config Snippets for reference:

JavaScript

// Mic Constraints
{ noiseSuppression: !willUseKrisp, echoCancellation: true, autoGainControl: true }

// Publish Defaults
dtx: false, 
audioPreset: { maxBitrate: 48_000 }, 

Any guidance on tuning our React client or deciding on the Native path would be hugely appreciated.

This is a bit beyond my area of expertise so I am relying to Claude for some of this.

Great follow-up, and the telemetry you shared makes each of these answerable pretty concretely. Those two profiles — GCC students with 10s RTT spikes and 91% loss windows, versus a US student with flawless transport stats who still hears lag — are actually two completely different problems, and it’s worth being explicit that no single fix addresses both. Taking them in order.

Q1: Proactive relay vs. the reactive watchdog

Yes — flip it. For students you can identify as GCC (geo-IP at token-issue time is fine), pass iceTransportPolicy: 'relay' on initial mount and let the watchdog become a backstop instead of the primary mechanism.

The reasoning: your watchdog is correctly detecting a failure mode the SDK can’t fully see. A DPI-throttled UDP flow is a zombie — ICE doesn’t transition to failed because an occasional packet still gets through, so the built-in reconnect logic has nothing unambiguous to trigger on. Your flat-packet-counter trick (with dtx: false so silence can’t masquerade as death) is a legitimate detector for that. But look at what the user experiences with the reactive design: ~5–15s of working audio → throttle kicks in → 9 seconds of confirmed silence → key bump → full teardown and remount → several more seconds of reconnect. That’s 15–25 seconds of dead air per throttle event, in a tutoring session. Relay-from-start means the flow that gets whacked never exists.

Two refinements worth keeping from your current design:

  • Make relay sticky. When the watchdog does trip for a non-GCC user, don’t just recover that session — persist a per-user forceRelay flag so their next session starts on relay. Networks that whack UDP once will do it every time.
  • Keep the watchdog even for relay users, with a longer threshold. TURN/TLS over a hostile network can still die (e.g. NAT rebinding, carrier-grade NAT timeouts); a backstop that forces a clean reconnect is still worth having. But it should be firing rarely — if it fires often on relay sessions, that’s a signal worth posting about.
flowchart TD
    A[Token request] --> B{Geo-IP / ASN in<br/>known-hostile list?}
    B -->|yes| C[Mount with<br/>iceTransportPolicy: 'relay']
    B -->|no| D[Mount with default ICE]
    D --> E{Watchdog trips?}
    E -->|yes| F[Reconnect with relay<br/>+ persist sticky flag]
    F --> C
    E -->|no| G[Normal session]

One small note: with relay-first in place, you could return dtx to its default (true) for those users and save bandwidth — but your flatline detector depends on dtx: false, and on a link where you’ve already dropped the audio bitrate (see Q2) the DTX savings are modest. Keeping dtx: false for detector semantics is a reasonable trade.

Q2: Are the 48k/64k bitrates feeding the TCP queue?

They’re not the main culprit, but they’re not helping, and there’s a multiplier you may not have priced in: RED.

First, for reference — your manual values map exactly onto existing presets: 48_000 is AudioPresets.music (which is also the SDK default) and 64_000 is AudioPresets.musicStereo. So you haven’t actually deviated from defaults much; you’ve just written them out by hand.

The multiplier: red: true is on by default, and redundant encoding roughly doubles the audio payload on the wire (each packet carries the previous frame too). So your student is publishing on the order of ~100 kbps of audio and the teacher ~130 kbps, before a single video packet. On a link that DPI has throttled to a few hundred kbps — or less — that’s real money. For your relay/GCC cohort, drop to AudioPresets.speech (24 kbps, ~50 kbps with RED): it’s tuned for voice and the perceptual difference for speech is small. Keep RED on — on a lossy path it’s doing far more good than its bandwidth costs.

But the honest answer about your 10-second RTTs: that’s not audio, that’s bufferbloat, and the whale is video. A 10s RTT means multiple seconds’ worth of bytes are sitting in TCP socket buffers and middlebox queues because the sender keeps offering more than the throttled rate. Two things in your current config make that worse:

  • Default video capture is 720p, and you have simulcast: true — so the publisher is encoding and uplinking multiple spatial layers. In a 1:1 session, only one is ever consumed. Either enable dynacast: true in your room options (it pauses layers no one subscribes to — it’s off by default), or disable simulcast for the constrained cohort and publish a single modest layer with an explicit videoEncoding.maxBitrate.
  • WebRTC’s congestion controller adapts, but over a TURN/TCP path its feedback is mushy and it’s slow to find the true (DPI-imposed) ceiling — meanwhile the queue is already seconds deep and your audio is stuck behind it. The reliable fix is a hard sender-side ceiling below the throttle rate, not trusting adaptation to find it.

Rule of thumb for the GCC/relay cohort: speech-preset audio + RED, single video layer capped aggressively (or camera off with your existing quality-based downgrade), total uplink budget in the low hundreds of kbps. That’s the configuration under which a TCP path stops turning loss into 10-second stalls.

Q3: Perfect transport stats, but the user hears “lag”

This is the most important distinction in your whole post: connection-quality telemetry measures the transport, not the mouth-to-ear path. RTT, jitter, and loss are sampled between the client and the edge. None of it sees what happens after the packet arrives — and “laggy audio” with clean transport stats almost always lives there. No, the standard telemetry does not capture device-level latency; you have to instrument it yourself. The usual suspects, roughly in order of likelihood:

  1. Jitter-buffer growth. The browser’s adaptive jitter buffer (NetEQ) inflates its target delay after jitter spikes — your US student’s max jitter was 127ms, which is plenty to trigger growth — and it deflates much more slowly than it inflates (it shrinks by playing audio slightly fast, which it does conservatively). Result: a call that accumulates 300–500ms+ of playout delay after a rough patch and holds it for minutes. That is experienced exactly as “the other person is laggy,” and it shows up in zero of the stats you listed. Check jitterBufferDelay / jitterBufferEmittedCount (a running average, so track its trend) on the inbound-rtp stats — you can get the raw report from the track’s receiver, and track.getReceiverStats() in livekit-client also gives you concealedSamples / concealmentEvents, which are the closest thing to a direct “audible glitch” counter.
  2. Bluetooth. A student on standard Bluetooth headphones is adding ~100–300ms of device latency, and if the microphone is also on Bluetooth, the headset drops into a handsfree profile with its own capture path and quality penalty. Log MediaDeviceInfo.label for the active input/output devices with your telemetry — you’ll be able to correlate “laggy” reports with AirPods/buds in about a day.
  3. CPU-starved processing. You’re stacking browser AEC + AGC (+ Krisp for some users) — on a weak laptop, that pipeline plus encode can fall behind in bursts. totalProcessingDelay on inbound-rtp is worth logging alongside the jitter-buffer numbers. Background-tab throttling does the same thing if the student switches tabs.
  4. The other leg. In a 1:1 call, the student’s perceived lag is teacher-uplink + SFU + student-downlink + both device chains. If the “flawless” stats you quoted are from the student’s client only, pull the teacher’s session for the same time window before concluding the network was clean.

For a “0 glitches” mandate, I’d make concealmentEvents per minute your primary quality KPI rather than poor sample percentage — it counts the moments a human actually heard something break, which is what your stakeholders mean by “glitch.”

Q4: Will going native fix any of this?

For the firewall/DPI problem: no, and it’s worth killing this hope precisely. The React Native SDK sits on react-native-webrtc, which is the same libwebrtc engine Chrome uses — same ICE, same STUN/TURN behavior, same UDP ports, same TURN/TLS-on-443 fallback, same congestion control. On the wire, a native app’s packets are indistinguishable from your web app’s packets, and the ISP’s DPI box treats them identically. There is no browser-sandbox limitation involved in your GCC story — the browser is faithfully executing a connection through a network that throttles UDP. Native would be development complexity spent on the one layer that doesn’t change.

Where native does genuinely help is your Q3 bucket — the device-side audio pipeline:

  • Control of the OS audio session and access to hardware/platform AEC, which is meaningfully better than browser software AEC on mobile devices.
  • Sane Bluetooth routing instead of whatever the browser negotiates.
  • No tab-backgrounding/throttling; calls survive the app going to background.
  • Consistent behavior across your users instead of per-browser AEC/AGC quirks.

So the sequencing I’d recommend: ship relay-first + the Q2 bitrate discipline now (it’s a config change, and it targets your two worst cohorts directly), instrument jitter-buffer delay, concealment events, and device labels next — and only then decide on native, based on whether the residual complaints are device-pipeline problems (native helps) or network problems (it won’t). One honest expectation to set upstream: “literally 0 glitches” over consumer ISPs that actively attack the traffic isn’t a reachable target on any stack — but “glitches rare enough that nobody mentions them,” measured as concealment events per minute, very much is.

Thanks for this info @CWilson.
If you dont mind, do you have any technical team or engineer with whom we can discuss more closely, someone who has worked closely on video conferencing platforms/edtech built over Livekit? Actually this is my first time building something like this, so I am not sure about the fixes which you mentioned via claude will work or not because coincidently i have been using Replit for building and i have done quite investigation but the Replit Agent keeps on reiterating over same loop and i am afraid that reiteration might break the architecture. Thats why if i could get some information from an expert then i would be more than happy. Please don’t get me wrong, i really appreciate your efforts and i as a junior techie really look towards amazing people like you who are working on such cost efficient products, because i had a very tight budget and couldn’t afford enterprise level sdks like zoom sdk etc for video conferencing.

I will share this with the team to see if anyone has some advice.

I got some feedback from the team on this:

In addition to setting iceTransportPolicy: relay, we can potentially disable embedded TURN for your project (will be applied project-wide, though, unless you have a specific room name or participant identity pattern for participants from GCC where we can enable it specifically for those patterns). The embedded TURN is still UDP. Client side iceTransportPolicy: relay will include all TURN relay candidates. So, we will have to disable the embedded TURN advertisement on the server side so the client does not try it.

Looks like the student is doing simulcast. Does the student need to do simulcast ? Can they do a single quality layer or simulcast with lower bitrates?

For the US student, wondering if that is the result of the other end struggling to send data properly. Not sure where the other end is.

Are these sessions 1:1? Or multi-player?

Would be good to get some sessions to look at.Can you share some session ids (starts with RM_*) along with a disctiption of the issue you saw for that specific session?

These sessions are 1:1. Sharing few session links -

OIT_LMS | Session | RM_nhSmTYYaB554
the above sessionid is from a 1:1 session between teacher and student. The student complained that the audio was breaking a lot.

OIT_LMS | Session | RM_bN5wErCtxpAa - this session id is from the student connecting from USA.
Interestingly, none of the students complain anything about any issue if they take it up on zoom.

Thanks for this. I am passing that on to the experts.

This is not something we generally ask from folks since it is a little involved. But sounds like you already know how to do this. Can you provide client-side WebRTC logs and dumps as well?

That would help us a lot.

https://drive.google.com/drive/folders/1qokxbiMpUd97gzOk40vMJBmfQukImUm3?usp=drive_link

Hi. Here is the data, i have uploaded as csv format. Kindly let me know if you are able to access it or not. Thanks

@Faraan_Ahmed_Hashmi can you also grant access to me please, I requested access through Google drive

OIT_LMS | Session | RM_nhSmTYYaB554
the above sessionid is from a 1:1 session between teacher and student. The student complained that the audio was breaking a lot.

For this session, it looks like simulcast is enabled for both camera and screen share - the suggestion is to disable simulcast for the screen share (i.e. only a single quality screen share is published, and the camera can fall back to lower quality if need be)

OIT_LMS | Session | RM_bN5wErCtxpAa - this session id is from the student connecting from USA.
Interestingly, none of the students complain anything about any issue if they take it up on zoom.

It is less clear what is going on here, possibly mobile issues.


Thank you for sharing those files, but there was nothing revealing in those. What would help, but it would require it to be done on the user’s end, would be to provide a webrtc internal dump. This is an external video, but shows the process: https://youtu.be/Qh2kEsAVqZ8

oh so do i need to run the test from the end user? That might be bit difficult because we are in a B2C environment.
Is there any custom function or layer that we can write which runs in background and collects the required webrtc dump?

Is there any custom function or layer that we can write which runs in background and collects the required webrtc dump?

Unfortunately not. Understood this was a tall ask as I appreciate the difficulty.