Track Egress concurrency per instance + private subnet deployment in OCI Kubernetes

I have a couple of questions regarding self-hosted LiveKit Egress, specifically Track Egress.

1. Multiple Track Egress jobs on a single Egress instance

The Egress Helm documentation says:

“Each instance can record one room at a time.”

I’m trying to understand whether this limitation applies to Track Egress specifically, or mainly to room/composite recording.
Our use case is Track Egress. We record each participant’s audio and video tracks separately and then use our own recording merge service to combine the tracks into a final MP4.
I have currently deployed one Egress instance, and I have successfully started Track Egress recordings for multiple rooms simultaneously using that single Egress instance. So far, it appears to be working correctly.

For example:

Egress Instance
 ├── Room A
 │    ├── Track Egress - User 1 Audio
 │    ├── Track Egress - User 1 Video
 │    └── Track Egress - User 2 Audio/Video
 │
 ├── Room B
 │    ├── Track Egress - User 1 Audio
 │    └── Track Egress - User 1 Video
 │
 └── Room C
      └── Multiple Track Egress jobs

My questions are:

  1. Does the “one room per instance” statement apply to Track Egress, or is it primarily intended for RoomComposite/Web/other resource-intensive Egress types?
  2. Is it officially supported to run multiple Track Egress jobs concurrently on the same Egress instance, including jobs belonging to different rooms?
  3. If yes, are there any limitations or recommended maximums for the number of concurrent Track Egress jobs per instance?
  4. Can multiple Track Egress jobs from the same room also run on the same Egress instance?
  5. Are there any concerns with running many simultaneous Track Egress jobs on one instance in terms of CPU, memory, network bandwidth, file I/O, or GStreamer resources?

2. Running Egress in a private subnet on Oracle Cloud Kubernetes

We are running LiveKit on Oracle Cloud OKE using VCN-native Pod networking.
Our LiveKit Server and LiveKit SIP deployments currently use:


hostNetwork: true

and are running on nodes/subnets with public Internet connectivity because of the networking requirements of LiveKit WebRTC/SIP.

Our question is:

  1. Can self-hosted LiveKit Egress run in a private Pod subnet in OCI OKE?

  2. Does Egress require a public IP/public subnet?

  3. Does Egress require hostNetwork: true, like the LiveKit Server?

  4. Can Egress run with normal Kubernetes Pod networking (hostNetwork: false)?

  5. If running Egress in a private subnet, is outbound Internet access through an OCI NAT Gateway sufficient for S3 and other external connections?

  6. Are there any specific inbound/outbound ports that must be allowed for Egress?

  7. Does Egress require direct UDP/TCP access to the node similar to the LiveKit Server, or does it establish its media connection through the LiveKit WebSocket/RTC connection without requiring hostNetwork?

    Any guidance on the recommended networking architecture for self-hosted Egress on OKE/OCI would be greatly appreciated.
    Thanks!

Good questions, and the two halves have clean answers. Both are documented, just in places that contradict each other if you only read one.

Part 1: Track Egress concurrency

Where that “one room at a time” line comes from

It appears in the Helm section of the self-hosting page, as guidance for picking replicaCount. The Requirements section on that same page says something different and more precise:

An egress worker may process one or more jobs at once, depending on their resource requirements. For example, a TrackEgress job consumes minimal resources because it doesn’t need to transcode. Consequently, hundreds of simultaneous TrackEgress jobs can run on a single instance.

So the Helm sentence is simplified advice aimed at RoomComposite users, where it is accurate because each RoomComposite job spawns a full Chrome instance and can consume 2 to 6 CPUs on its own. It is not a constraint on Track Egress. Your observed behaviour is the documented behaviour, not luck.

The architectural reason: RoomComposite and Web requests launch Chrome against a web template. Track and TrackComposite requests use the Go SDK directly, with no browser, no Xvfb, and no PulseAudio. A pure Track Egress writing the track in its native container does not transcode at all.

Your five questions

1. Does it apply to Track Egress? No. It is RoomComposite and Web guidance.

2. Is multi-room on one instance supported? Yes. There is no room affinity anywhere in the scheduler. Requests are distributed over Redis pub/sub and each worker accepts or declines based on its own resource accounting. Nothing binds a worker to a room, so Rooms A, B and C landing on one instance is normal operation, not an accident you are getting away with.

3. Recommended maximum? There is no hard job count. Admission is governed by the cpu_cost block, with these defaults from the Egress README:

cpu_cost:
  room_composite_cpu_cost: 3.0
  web_cpu_cost: 3.0
  track_composite_cpu_cost: 2.0
  track_cpu_cost: 1.0

Worth flagging a tension here: read literally, track_cpu_cost: 1.0 against a 4 CPU pod caps you around 3 or 4 concurrent track jobs, which does not square with “hundreds”. Rather than trusting either number, watch this log line from stats/monitor.go on your own build:

INFO egress stats/monitor.go:139 cpu available: 4.000000 max cost: 0.300000

That tells you what your instance actually thinks it can take. If you start seeing requests declined earlier than you expect, set track_cpu_cost explicitly (something like 0.1) rather than relying on the default. Do not size your fleet off the README number without measuring.

4. Multiple jobs from the same room on one instance? Yes. Each egress joins the room as its own participant with kind = EGRESS and subscribes only to the tracks it needs. A TrackEgress for one audio track subscribes to exactly that track. Five jobs in Room A means five subscribe-only participants, which is no different from five extra viewers.

5. Real resource concerns. CPU is the one people worry about and the one that will not bite you. In rough order of what actually will:

  • Disk, not CPU. Every job writes to local disk before upload. Concurrent long sessions multiply fast, and running out of ephemeral storage fails late, after you have already recorded the content. Size the volume for worst-case concurrent jobs x session length x bitrate, and set backup_storage so failed uploads land somewhere recoverable instead of disappearing.
  • Bandwidth. This is your real ceiling. Every track job is a live subscription. Thirty video tracks at 2 Mbps is roughly 60 Mbps sustained into one pod, plus upload bursts to object storage on top. You will saturate the pod’s network long before you saturate 4 CPUs.
  • Blast radius. One pod is one failure domain. An OOM, an eviction, or a routine node drain kills every in-flight recording across all rooms simultaneously. Because track egress is not CPU-bound you will naturally be tempted to run a single replica, which is a materially worse availability posture than RoomComposite users get by accident. Run at least 2 replicas and add a PodDisruptionBudget. A kubectl drain during a maintenance window will otherwise destroy live recordings with no warning.
  • Session limits. Check session_limits.file_output_max_duration, which defaults to 1h in the self-hosted config. Long sessions end with EGRESS_LIMIT_REACHED rather than an error you would notice in testing.
  • Autoscaling metric caveat. Be careful with livekit_egress_available. Despite the docs describing it as CPU-based, it was reported as returning 1 only when the instance has zero active requests, meaning an HPA on it scales out on the very first track job even though the pod could handle hundreds more. See livekit/egress#1110. That issue is now closed so it may be fixed on current versions, but verify on whatever tag you are running before you build an HPA on it. For a track-only workload, scaling on bandwidth or a custom metric is a better fit anyway.

Part 2: Egress in a private subnet on OKE

Short answer to all seven: yes it works, no public IP, no hostNetwork, NAT Gateway is sufficient, and no inbound ports are required.

The reason is that Egress has a fundamentally different network shape from LiveKit Server and SIP. It is a subscribe-only WebRTC client, and it never receives an inbound API request. It picks up work from Redis pub/sub and initiates every connection outbound. It does not advertise host candidates that remote peers dial into, which is the entire reason Server and SIP need hostNetwork: true and a routable node IP. Egress needs none of that.

1. Private pod subnet? Yes.
2. Public IP or public subnet? No.
3. hostNetwork: true? No. Do not set it.
4. Normal pod networking (hostNetwork: false)? Yes, this is the correct configuration. VCN-native pod networking is fine.
5. NAT Gateway sufficient for S3? Yes, with one improvement below.
6. Inbound/outbound ports?

Inbound: none from outside the cluster. Optionally allow in-cluster scraping of health_port and prometheus_port if you have configured them.

Outbound:

Destination Purpose Path
Redis job dispatch, must be the same Redis as livekit-server intra-VCN
livekit-server ws_url signaling intra-VCN
livekit-server media (UDP mux or port range, plus 7881/TCP fallback) the actual media subscription intra-VCN
Object storage uploads NAT Gateway or Service Gateway
DNS resolution intra-VCN

7. Direct UDP/TCP node access? Not to Egress. But Egress does need to reach the LiveKit Server’s media ports, and that is the one thing that genuinely trips people up on this exact setup. Details below.

The gotcha to check before you deploy

Your LiveKit Server runs with hostNetwork: true and almost certainly use_external_ip: true, which means it advertises its public IP as an ICE candidate. An Egress pod sitting in a private subnet and routing out through a NAT Gateway will then try to reach that public IP. NAT hairpinning back to a public IP belonging to a node inside the same VCN frequently does not work on OCI.

The symptom is specific and easy to misdiagnose: the WebSocket connects fine, the egress job starts and reports as active, and then you get zero-length files, empty recordings, or a media timeout. It looks like an Egress bug and it is a routing problem.

To avoid it:

  • Point Egress ws_url at the internal ClusterIP or headless Service for livekit-server rather than the public endpoint, so signaling stays inside the VCN.
  • Confirm your security lists or NSGs allow the pod subnet to reach the node subnet on the LiveKit media ports (your UDP mux port or port_range_start/end) and on 7881/TCP. This is intra-VCN traffic, so it is a security list rule, not anything to do with the NAT Gateway.
  • Verify which candidate pair actually wins on a test job before you trust it. If everything is landing on 7881/TCP you are working but relaying unnecessarily.

Two OCI-specific recommendations

  • Use a Service Gateway, not the NAT Gateway, for OCI Object Storage. Continuous track-egress upload volume through a NAT Gateway is both slower and billed. A Service Gateway keeps that traffic on the OCI backbone. If you are using OCI Object Storage via its S3-compatible endpoint, set endpoint in the Egress storage.s3 config accordingly. Keep the NAT Gateway for anything else that needs general internet access.
  • Give the pod real scratch space. By default container ephemeral storage comes off the node boot volume, which is not sized for hours of concurrent media. Mount an emptyDir with a sizeLimit or a block volume PVC and point Egress at it, and make sure the mount is writable by the non-root user, since Egress does not run as root.

References: