Setup: voice agents on livekit-agents 1.4.6 / livekit 1.1.2 / livekit-api 1.0.7 (Python 3.11), inbound SIP telephony, LiveKit Cloud (project p_3tqm7ro6kbs). At job entry we start a room-composite audio egress (RoomCompositeEgressRequest, audio_only=True, dual-channel) with an S3 file output, and let it run for the whole call.
What happened (one inbound call): a normal ~6 min 14 s call was recorded by egress EG_NbMbijFpqtZz. Partway through, the egress participant migrated twice within ~5 seconds and never rejoined, the egress finalized early (~1 min 52 s in), and its egress_ended webhook fired while the call was still active. Net result: the saved recording contains only the first 1 min 52 s of a 6 min 14 s call — ~4 min 20 s (≈70%) is missing — and the egress output was also incomplete.
Room session events (LiveKit dashboard, UTC):
Time (UTC)
Participant
Event
16:45:37
—
Room created
16:45:37
sip_+1719XXXXXXX
joining → active
16:45:37–38
agent-AJ_N852nz42MQcc
joining → active
16:45:38
EG_NbMbijFpqtZz (egress)
joining → active
16:46:20
EG_NbMbijFpqtZz
left (MIGRATION) → joining → active
16:46:25
EG_NbMbijFpqtZz
left (MIGRATION) — no rejoin afterward
16:52:05
sip_+1719XXXXXXX
left (CLIENT_INITIATED)
16:52:19
agent-AJ_N852nz42MQcc
left (ROOM_DELETED) → Room ended → API.DeleteRoom
So the egress stopped ~1 min after the second migration and finalized at duration_ms=111880 (1 min 52 s), even though the call ran until ~16:51:53. There was no second/complete egress — the early-finalized partial file is the only recording.
@darryncampbell - I don’t think the issue is related to the incident. My hunch is the above issue is related to the post I raised a few weeks ago. BTW still no root cause identified for that too.
Will wait on your further findings for these issues.
Hey @Zaheer_Abbas - thanks for the report as always! I took a look - the issue you experienced was caused by the incident @darryncampbell linked. Internal routing algorithm routed egress participant to a different region. Agent and the caller were in the same region (different than egress) so the live communication between them kept working and was unaffected. Apologies for the inconvenience - this case will definitively inform our efforts to build more robust retry logic. We will also check to see if we could make this signaling better so that if the recording didn’t end naturally you also get some feedback that could be checked.
If this is related to the incident, how can we find if more such cases occurred. There is no way in livekit currently to filter what calls or sessions ran into issue during an incident and how much were our project impacted - Live Sessions, Egress, Webhooks etc.
It would be really helpful to add that in the dashboard.
