Egress failed with "Egress Timeout"

Egress / Ingress
,
1

Hi, we have a recording that’s failed with egress timeout.

Any idea what may have gone wrong? seems to be within all limits and quotas.

I know there was talk about backups of files when errors occur. Any chance this file is retrievable?

Thanks

2

Can you share the session ID and egress ID?

3

EG_iEvCs28fAePx and RM_oeFXWsCiLKru

Thanks

4

@Milos_Pesic thoughts?

5

I’ve asked if there is a way to recover the file. I am waiting to hear back.

6

Any news on this? I’ve got a user here with fingers and toes crossed!

7

Sorry for the delay in getting back to you on this. Sorry to say, there is no way to recover this recording.

8

Ah ok, no worries.

Any thoughts for the reason for the error?

9

Hey! Saw your last question about the timeout was still open.

When an Egress Timeout happens without an obvious configuration error, it almost always comes down to a transient CPU or RAM spike during encoding.

Real-time video rendering and encoding (especially multi-participant room composites) is incredibly heavy on system resources. If the worker instance hits a momentary resource bottleneck or encoding buffer freeze, the process stalls, stops sending heartbeats, and the orchestrator eventually force-kills it with a timeout error.

Because cloud instances run in ephemeral sandboxes, once that worker is killed, any un-uploaded temp files are flushed from disk, which is why the recording isn’t retrievable.

A couple of quick tips to help prevent it on future runs:

Lightweight Layouts: If using custom room composite templates, simplify the DOM/CSS and lower the target render resolution slightly to keep CPU overhead low.

Use Track Egress: If you just need the raw audio/video and don’t strictly require a styled composite grid, direct Track Egress uses a fraction of the encoding resources.

Verify Storage Bucket Region: Ensure your destination S3/GCP bucket is in the closest geographical region to minimize upload stalls when closing out the stream. Just some thoughts, and hopefully it will help you out. Best of Luck.

I got to add this note: If you were using RoomComposite, spinning up headless Chromium to render and re-encode multi-video DOM layouts is extremely resource-heavy—a brief CPU spike during rendering will freeze the browser loop and trip the timeout.
I mention this, because I had to move to trackComposite, and avoid firing up the headless chrome browser. I must use every resource I have to the max, so I account for anything needed, is running as efficiently as I can make it. Keep at it, you will get it right.

10

While some points from @DoC_DiD_iT above are correct - we need a set of small corrections. It’s true that once something goes wrong with the process (e.g underlaying hardware failure) currently opened files are being flushed and the recording to that point isn’t retrievable.
However - on cloud - we have tight control over resources and admissions of new requests as well as robust autoscaling process and default templates almost never cause process to be killed due to high resource usage. What happened in this particular case was underlaying hw failure and we are working at detecting these earlier and preventing similar issues in future. That being said - I would say no need to to use track egress or lightweight layouts (unless custom templates with heavy rendering) as mitigation measures - it’s not the resource issue.

One thing which could help preserving data in cases like this is using HLS instead of files when possible - segments and playlist updates are going to be continuously updated, even on hw failures the existing portion of the recording will be available.

11

Thanks for the clarification.

Just to clarify (and for anyone stumbling across this in the future) it feels like the options are:

  • Leave things as they are - improvements are being made at detecting hardware failuires
  • Export as individual tracks - then post-process to put them together again - might reduce load slightly, but a hardware failure in egress would likely still result in data loss
  • Stream as HLS to S3 - then once complete run a post-process to combine all of the segments so they are downloadable by users for playback and distribution - has the benefit that if anything crashes mid-recording / processing you’ve always got the segments up to that point

Can you think of any downsides (practical or financial) to the HLS approach? Above the fact that we’d have to run a process on a server somewhere to convert the files?

Thanks

12

Hey @Ryk_BilateralBase thanks for bringing up these points - answering inline:

  • Leave things as they are - improvements are being made at detecting hardware failuires

This is correct.

  • Export as individual tracks - then post-process to put them together again - might reduce load slightly, but a hardware failure in egress would likely still result in data loss

Don’t do this - it’s going to be impossible to post-process them together, mixing process is stateful and quite sophisticated, accurate post-process mixing won’t be possible

  • Stream as HLS to S3 - then once complete run a post-process to combine all of the segments so they are downloadable by users for playback and distribution - has the benefit that if anything crashes mid-recording / processing you’ve always got the segments up to that point

No significant downsides. Financially it’s near-neutral: the media is encoded once either way. Two small practical caveats:

  1. Only completed segments are uploaded, so a hard crash can still lose the last few seconds (up to one segment duration, plus anything queued for upload).
  2. The playlist in storage is updated periodically rather than per segment, so after a crash it may be slightly stale - have your post-processor list the segment objects in the bucket rather than trusting the playlist, and tolerate a missing EXT-X-ENDLIST tag.

The team will likely build something like this for cloud anyways as an opt-in feature, even for cases when a file output is specified - with post-processing done on our side. Can’t give an exact timeline yet but it’s on the horizon.

One more option: specify both a file and an HLS output on the same egress - the media is still only encoded once, and you only need post-processing for the rare case where the file didn’t get uploaded.

13

Assume there would be a duplicate cost for both uploads? (i haven’t looked at the egress pricing for a while)

14

Thanks for jumping in and clarifying how LiveKit Cloud handles host node failures under the hood!

That makes total sense regarding the container flushing—and that HLS tip is huge. Streaming raw segments out continuously to avoid losing data on unexpected host drops/timeouts is a great best practice for anyone needing fail-safe recordings.

Appreciate the insight and the hard work you guys are doing on the platform! I am working with a old Chevy for a server, and you guys are running a Bentley, so I sure don’t knock ya. Keep up the good work

15

That’s awesome news that an automated HLS fallback + cloud post-processing pipeline is on the roadmap! Having the backend handle segment stitching when a container drops will be a huge safety net for file-based recordings.

Anyone who has spent late nights watching /tmp fill up on a media server, hunting down orphaned .ts chunks, or watching Chromium chew through host CPU cores knows the real pain of real-time egress!

Appreciate the thorough breakdown and tips on listing bucket objects over playlist tags—super helpful context.