Egress failed with "Egress Timeout"

Egress / Ingress
,
1

Hi, we have a recording that’s failed with egress timeout.

Any idea what may have gone wrong? seems to be within all limits and quotas.

I know there was talk about backups of files when errors occur. Any chance this file is retrievable?

Thanks

2

Can you share the session ID and egress ID?

3

EG_iEvCs28fAePx and RM_oeFXWsCiLKru

Thanks

4

@Milos_Pesic thoughts?

5

I’ve asked if there is a way to recover the file. I am waiting to hear back.

6

Any news on this? I’ve got a user here with fingers and toes crossed!

7

Sorry for the delay in getting back to you on this. Sorry to say, there is no way to recover this recording.

8

Ah ok, no worries.

Any thoughts for the reason for the error?

9

Hey! Saw your last question about the timeout was still open.

When an Egress Timeout happens without an obvious configuration error, it almost always comes down to a transient CPU or RAM spike during encoding.

Real-time video rendering and encoding (especially multi-participant room composites) is incredibly heavy on system resources. If the worker instance hits a momentary resource bottleneck or encoding buffer freeze, the process stalls, stops sending heartbeats, and the orchestrator eventually force-kills it with a timeout error.

Because cloud instances run in ephemeral sandboxes, once that worker is killed, any un-uploaded temp files are flushed from disk, which is why the recording isn’t retrievable.

A couple of quick tips to help prevent it on future runs:

Lightweight Layouts: If using custom room composite templates, simplify the DOM/CSS and lower the target render resolution slightly to keep CPU overhead low.

Use Track Egress: If you just need the raw audio/video and don’t strictly require a styled composite grid, direct Track Egress uses a fraction of the encoding resources.

Verify Storage Bucket Region: Ensure your destination S3/GCP bucket is in the closest geographical region to minimize upload stalls when closing out the stream. Just some thoughts, and hopefully it will help you out. Best of Luck.

I got to add this note: If you were using RoomComposite, spinning up headless Chromium to render and re-encode multi-video DOM layouts is extremely resource-heavy—a brief CPU spike during rendering will freeze the browser loop and trip the timeout.
I mention this, because I had to move to trackComposite, and avoid firing up the headless chrome browser. I must use every resource I have to the max, so I account for anything needed, is running as efficiently as I can make it. Keep at it, you will get it right.

10

While some points from @DoC_DiD_iT above are correct - we need a set of small corrections. It’s true that once something goes wrong with the process (e.g underlaying hardware failure) currently opened files are being flushed and the recording to that point isn’t retrievable.
However - on cloud - we have tight control over resources and admissions of new requests as well as robust autoscaling process and default templates almost never cause process to be killed due to high resource usage. What happened in this particular case was underlaying hw failure and we are working at detecting these earlier and preventing similar issues in future. That being said - I would say no need to to use track egress or lightweight layouts (unless custom templates with heavy rendering) as mitigation measures - it’s not the resource issue.

One thing which could help preserving data in cases like this is using HLS instead of files when possible - segments and playlist updates are going to be continuously updated, even on hw failures the existing portion of the recording will be available.

11

Thanks for the clarification.

Just to clarify (and for anyone stumbling across this in the future) it feels like the options are:

  • Leave things as they are - improvements are being made at detecting hardware failuires
  • Export as individual tracks - then post-process to put them together again - might reduce load slightly, but a hardware failure in egress would likely still result in data loss
  • Stream as HLS to S3 - then once complete run a post-process to combine all of the segments so they are downloadable by users for playback and distribution - has the benefit that if anything crashes mid-recording / processing you’ve always got the segments up to that point

Can you think of any downsides (practical or financial) to the HLS approach? Above the fact that we’d have to run a process on a server somewhere to convert the files?

Thanks