Drift between audio and video in the participant recording egress livekit cloud

Egress / Ingress
1

I am trying to record egresss for participant recordings. no matter what i try, there is always drift between audio and video on the livekit side for the participant recording.i am using the giiven instruction but there is always a drift. any fixes or suggestions? it is impacting our prod pdocasting capabilities

2

I’m not sure about your setup, but perhaps moving to track composite egress would help? Participant & TrackComposite egress | LiveKit Documentation

3

I’m using TrackComposite and having this same issue. Logs show drift. I’m pushing via whip ingress key and url and use egress TrackComposite to grab the audio and video tracks. But, I’m getting a lot of drift.

It was working great, then started getting it. All 3 are using the latest versions

4

Do you know which version it stopped working in?

5

@arslan.javaid @Donald_DOC_OConnor Just checking - do you use LK Cloud services or host egress yourself?

In order to be able to help we would need some info:

  • if you use LK cloud - egressIDs of affected recordings would be good to have so we can check, ideally a sample of recording exhibiting the problem if you are able to share it.
  • if you self host - debug service logs, service configuration (stripped of sensitive info) and exact version of the egress image used

Thanks,
Milos

6

While the team gathers the data they asked for, two characterization questions often narrow A/V drift quickly:

  • Is the drift a fixed offset (audio and video start out misaligned but stay that way), or progressive (the gap grows over the recording’s duration)? Fixed offset points at session-start timestamp handling; progressive points at sample-rate mismatch or accumulated jitter compensation. Worth measuring on one affected recording.
  • @Donald_DOC_OConnor: does the drift show up only on WHIP-ingested tracks, or also when the audio/video is published directly from a client SDK (browser/mobile)? If WHIP-only, the issue is scoped to the WHIP >> SFU >> egress timestamp path, which is much easier to triage.

Both answers are something you can produce without Cloud access, and they sharpen what the team is looking for.

7

The drift was in the hardware encoder clock of the LiveKit egress service, not specific to WHIP. Any track source (WHIP, RTMP, browser SDK) going through egress would exhibit the same drift. I fixed it by enabling audio_tempo_controller in my egress.yaml, with: adjustment_rate: 0.08, which imperceptibly adjusts the audio playback speed to keep it locked to the video timestamp. I started with: 0.02, and adjusted. So, this is something others may need to play with, to get theirs corrected.

8

@DoC_DiD_iT, Nice, that lines up with the progressive-drift branch from above: an encoder-clock offset that grows is the sample-rate/clock case, and audio_tempo_controller is the intended fix for it. It’s the documented “audio tempo adjustments for compensating PTS drift” knob, with adjustment_rate as the dial you tuned [ livekit/egress pkg/config/base.go ]. Same block has a sibling, enable_sync_engine (“Chrome-inspired sync engine for improved cross-participant alignment and A/V sync”), worth a try if you’d rather use the newer sync path than tempo nudging.

One caveat for anyone landing here from the title: both are egress.yaml settings, so they only apply if you self-host egress. On LiveKit Cloud you can’t set them, so the original Cloud report still needs the egressIDs the team asked for, not this knob.

9

thank you. I checked 3 or 4 times, and didn’t see a reply. And, I thought I checked regularly. My last comment was what fixed it for me, and I’m going to go back into my code and check what you mentioned at the end. I am sure you correct, as my guest was most likely on an iPhone, causing issues with Turn and Ice Connections. Though, after my fix, it work great.

It ran smoothly ever since. I’ve successfully ran 5 interviews, and going to push it to the max, when I multicast my next interviews to my social platforms, in a couple days. I’m waiting on my Fiber to be connected. A must for any media. And, what I use fiber in my main office. Though, my home office is where I work from mostly. Which, was att air, and running OBS studio, on a Laptop. :slight_smile: Amazingly, all works well on it, but I keep it small.

Now, it is self hosted on my Server in a datacenter. All, but OBS Studio. Maybe too much info for prying eyes, but we all know the feeling of having everything working correctly.

I will add: Not sponsored by, but Powered By LiveKit & DOC’s media. :wink: as all frontend, and backend code is mine. Though, the engine sitting in that shinny car, is LiveKit. Thank you guys for providing this.

This is closed, and I highly recommend LiveKit. Let them run this in their Cloud for you, as they know the best solutions for their users. Self hosting takes a lot of time, and they will cut that out for you. :wink:
10

We have reproduced what appears to be the same progressive drift described in this thread.

Our browser preview and the LiveKit room remain synchronized indefinitely. The drift only appears in the YouTube RTMP output and grows progressively (~1 second after approximately 9 minutes).

We have already ruled out capture hardware by testing both a USB HDMI capture device and Apple’s Continuity Camera.

Because we are using LiveKit Cloud, we cannot configure audio_tempo_controller or enable_sync_engine.

Can someone confirm whether the current LiveKit Cloud Egress workers enable either of these synchronization mechanisms?

11

Those are both disabled. Can you please provide an Egress ID that shows the drift, it starts with EG_, and we can take a look.

12

Thank you Darryn - here is the latest one - EG_iAW49BqYWh4c

13

Thanks for the ID. I can’t see an obvious root cause I’m afraid, nor have I seen this before. I might need to ask @Milos_Pesic to please take a look if you get a chance?

15

Hey @pastortim - thanks for the details. I checked logs we have for the session id you provided. Based on logs streams were kept in sync all the way to the end with only minor adjustments < 10ms, which is far away from what you observed. Could you reproduce the issue with a file output attached alongside RTMP streams? If a/v sync is fine for the file recording - that will point us to something inside or downstream of the RTMP output. Otherwise - if we see the same issue inside a file recording that would indicate an issue with egress sync logic. In any case that would be great signal to have if possible.