Agent voice breaking in S3 composite recording but clean in LiveKit Observability

Getting Started
1

Environment

  • SDK: livekit-agents (Python, latest)
  • TTS: ElevenLabs eleven_flash_v2_5
  • STT: Groq Whisper-large-v3-turbo
  • Noise cancellation: ai_coustics QUAIL_VF_L @ enhancement_level=0.8 (web/WebRTC calls)
  • Call type: WebRTC browser
  • Recording: RoomCompositeEgress → MP3 → AWS S3

Setup

Two completely independent deployments for worker — separate LiveKit accounts, separate
servers, same codebase:

  • Server 1 → LiveKit Account 1 → multiple accounts → recording works perfectly
  • Server 2 → LiveKit Account 2 → one specific account → recording is breaking

Problem

On Server 2, the agent voice is audibly choppy/breaking in the composite
recording fetched from S3. When we listen to the same session in LiveKit
Observability the audio is perfectly clean.

  • LiveKit Observability (agent track) — clean
  • S3 RoomCompositeEgress recording (MP3) — agent voice is breaking

Server 1 with the same code and same egress configuration produces clean
recordings. Only Server 2 is affected.

Is there any account-level configuration (plan, egress limits, media server
assignment) that could cause composite recordings to have audio artifacts
while the individual agent track in Observability remains clean?

Any insight appreciated.

2

Do you have a session ID and an example of the audio from egress and insights? For insights, you can just share with LiveKit staff for that session.

3

Hi @CWilson

I’ve attached the LiveKit session ID along with the S3 recording clip where the agent’s voice is breaking.

Livekit SessionId: RM_vUZd7J95UiCj

The agent’s voice breaks multiple times throughout the conversation in the S3 recording, while the corresponding LiveKit session audio is clean and does not exhibit the issue.

I also tested with other users using the same LiveKit account and observed the same behavior. Since this issue is consistently occurring only with this LiveKit account, it seems likely that the problem is account-specific rather than user-specific.

Could you please take a look and let me know if there’s anything unusual with this account or its recording pipeline?

4

Thanks for the session ID. I dug into it, and the good news is it’s not account-level. The root cause is CPU starvation on the Agent worker server itself (Server 2).

Comparing your S3 clip against the session audio, the breaks are moments where the agent published audio late: ~0.4s of silence gets inserted, and everything after stays shifted. That means the audio left the agent process behind realtime. Your agent logs confirm it, there are inference is slower than realtime warnings (from Silero VAD) at the exact timestamps the recording breaks.

Why Observability sounds clean: the audio samples are contiguous on the wire, just sent late. Observability reconstructs from timestamps, so the delay is invisible — but real-time consumers like the composite egress hear the gaps.

Your worker is on a t3.medium (2 vCPU, burstable), running QUAIL_VF_L at an enhancement_level of 0.8, plus VAD/STT/TTS. That’s the load causing the stalls.

Suggestions:

  1. Move the worker to a non-burstable instance with more cores (e.g., c-family, 4+ vCPU), or check CloudWatch CPUCreditBalance / CPU steal on Server 2.
  2. Try lowering the QUAIL enhancement_level; it’s the heaviest CPU consumer.
  3. Watch for inference is slower than realtime warnings — they accompany every audible break.

Quick way to confirm: point Server 2’s worker at Account 1, you’ll see the choppiness follows the server, not the account.