Good questions, and the two halves have clean answers. Both are documented, just in places that contradict each other if you only read one.
Part 1: Track Egress concurrency
Where that “one room at a time” line comes from
It appears in the Helm section of the self-hosting page, as guidance for picking replicaCount. The Requirements section on that same page says something different and more precise:
An egress worker may process one or more jobs at once, depending on their resource requirements. For example, a TrackEgress job consumes minimal resources because it doesn’t need to transcode. Consequently, hundreds of simultaneous TrackEgress jobs can run on a single instance.
So the Helm sentence is simplified advice aimed at RoomComposite users, where it is accurate because each RoomComposite job spawns a full Chrome instance and can consume 2 to 6 CPUs on its own. It is not a constraint on Track Egress. Your observed behaviour is the documented behaviour, not luck.
The architectural reason: RoomComposite and Web requests launch Chrome against a web template. Track and TrackComposite requests use the Go SDK directly, with no browser, no Xvfb, and no PulseAudio. A pure Track Egress writing the track in its native container does not transcode at all.
Your five questions
1. Does it apply to Track Egress? No. It is RoomComposite and Web guidance.
2. Is multi-room on one instance supported? Yes. There is no room affinity anywhere in the scheduler. Requests are distributed over Redis pub/sub and each worker accepts or declines based on its own resource accounting. Nothing binds a worker to a room, so Rooms A, B and C landing on one instance is normal operation, not an accident you are getting away with.
3. Recommended maximum? There is no hard job count. Admission is governed by the cpu_cost block, with these defaults from the Egress README:
cpu_cost:
room_composite_cpu_cost: 3.0
web_cpu_cost: 3.0
track_composite_cpu_cost: 2.0
track_cpu_cost: 1.0
Worth flagging a tension here: read literally, track_cpu_cost: 1.0 against a 4 CPU pod caps you around 3 or 4 concurrent track jobs, which does not square with “hundreds”. Rather than trusting either number, watch this log line from stats/monitor.go on your own build:
INFO egress stats/monitor.go:139 cpu available: 4.000000 max cost: 0.300000
That tells you what your instance actually thinks it can take. If you start seeing requests declined earlier than you expect, set track_cpu_cost explicitly (something like 0.1) rather than relying on the default. Do not size your fleet off the README number without measuring.
4. Multiple jobs from the same room on one instance? Yes. Each egress joins the room as its own participant with kind = EGRESS and subscribes only to the tracks it needs. A TrackEgress for one audio track subscribes to exactly that track. Five jobs in Room A means five subscribe-only participants, which is no different from five extra viewers.
5. Real resource concerns. CPU is the one people worry about and the one that will not bite you. In rough order of what actually will:
- Disk, not CPU. Every job writes to local disk before upload. Concurrent long sessions multiply fast, and running out of ephemeral storage fails late, after you have already recorded the content. Size the volume for
worst-case concurrent jobs x session length x bitrate, and set backup_storage so failed uploads land somewhere recoverable instead of disappearing.
- Bandwidth. This is your real ceiling. Every track job is a live subscription. Thirty video tracks at 2 Mbps is roughly 60 Mbps sustained into one pod, plus upload bursts to object storage on top. You will saturate the pod’s network long before you saturate 4 CPUs.
- Blast radius. One pod is one failure domain. An OOM, an eviction, or a routine node drain kills every in-flight recording across all rooms simultaneously. Because track egress is not CPU-bound you will naturally be tempted to run a single replica, which is a materially worse availability posture than RoomComposite users get by accident. Run at least 2 replicas and add a PodDisruptionBudget. A
kubectl drain during a maintenance window will otherwise destroy live recordings with no warning.
- Session limits. Check
session_limits.file_output_max_duration, which defaults to 1h in the self-hosted config. Long sessions end with EGRESS_LIMIT_REACHED rather than an error you would notice in testing.
- Autoscaling metric caveat. Be careful with
livekit_egress_available. Despite the docs describing it as CPU-based, it was reported as returning 1 only when the instance has zero active requests, meaning an HPA on it scales out on the very first track job even though the pod could handle hundreds more. See livekit/egress#1110. That issue is now closed so it may be fixed on current versions, but verify on whatever tag you are running before you build an HPA on it. For a track-only workload, scaling on bandwidth or a custom metric is a better fit anyway.
Part 2: Egress in a private subnet on OKE
Short answer to all seven: yes it works, no public IP, no hostNetwork, NAT Gateway is sufficient, and no inbound ports are required.
The reason is that Egress has a fundamentally different network shape from LiveKit Server and SIP. It is a subscribe-only WebRTC client, and it never receives an inbound API request. It picks up work from Redis pub/sub and initiates every connection outbound. It does not advertise host candidates that remote peers dial into, which is the entire reason Server and SIP need hostNetwork: true and a routable node IP. Egress needs none of that.
1. Private pod subnet? Yes.
2. Public IP or public subnet? No.
3. hostNetwork: true? No. Do not set it.
4. Normal pod networking (hostNetwork: false)? Yes, this is the correct configuration. VCN-native pod networking is fine.
5. NAT Gateway sufficient for S3? Yes, with one improvement below.
6. Inbound/outbound ports?
Inbound: none from outside the cluster. Optionally allow in-cluster scraping of health_port and prometheus_port if you have configured them.
Outbound:
| Destination |
Purpose |
Path |
| Redis |
job dispatch, must be the same Redis as livekit-server |
intra-VCN |
livekit-server ws_url |
signaling |
intra-VCN |
| livekit-server media (UDP mux or port range, plus 7881/TCP fallback) |
the actual media subscription |
intra-VCN |
| Object storage |
uploads |
NAT Gateway or Service Gateway |
| DNS |
resolution |
intra-VCN |
7. Direct UDP/TCP node access? Not to Egress. But Egress does need to reach the LiveKit Server’s media ports, and that is the one thing that genuinely trips people up on this exact setup. Details below.
The gotcha to check before you deploy
Your LiveKit Server runs with hostNetwork: true and almost certainly use_external_ip: true, which means it advertises its public IP as an ICE candidate. An Egress pod sitting in a private subnet and routing out through a NAT Gateway will then try to reach that public IP. NAT hairpinning back to a public IP belonging to a node inside the same VCN frequently does not work on OCI.
The symptom is specific and easy to misdiagnose: the WebSocket connects fine, the egress job starts and reports as active, and then you get zero-length files, empty recordings, or a media timeout. It looks like an Egress bug and it is a routing problem.
To avoid it:
- Point Egress
ws_url at the internal ClusterIP or headless Service for livekit-server rather than the public endpoint, so signaling stays inside the VCN.
- Confirm your security lists or NSGs allow the pod subnet to reach the node subnet on the LiveKit media ports (your UDP mux port or
port_range_start/end) and on 7881/TCP. This is intra-VCN traffic, so it is a security list rule, not anything to do with the NAT Gateway.
- Verify which candidate pair actually wins on a test job before you trust it. If everything is landing on 7881/TCP you are working but relaying unnecessarily.
Two OCI-specific recommendations
- Use a Service Gateway, not the NAT Gateway, for OCI Object Storage. Continuous track-egress upload volume through a NAT Gateway is both slower and billed. A Service Gateway keeps that traffic on the OCI backbone. If you are using OCI Object Storage via its S3-compatible endpoint, set
endpoint in the Egress storage.s3 config accordingly. Keep the NAT Gateway for anything else that needs general internet access.
- Give the pod real scratch space. By default container ephemeral storage comes off the node boot volume, which is not sized for hours of concurrent media. Mount an
emptyDir with a sizeLimit or a block volume PVC and point Egress at it, and make sure the mount is writable by the non-root user, since Egress does not run as root.
References: