500 Internal Server Error uploading session report to /observability/recordings/v0

Agents
,
1

Description

When agent observability/recording is enabled, the SDK fails to upload the session report at the end of a call with a 500 Internal Server Error from the LiveKit Cloud endpoint.

Error

aiohttp.client_exceptions.ClientResponseError: 500, message='Internal Server Error',
  url='https://<project>.livekit.cloud/observability/recordings/v0'

Full traceback:

File ".../livekit/agents/job.py", line 275, in _on_session_end
    await _upload_session_report(
File ".../livekit/agents/telemetry/traces.py", line 528, in _upload_session_report
    resp.raise_for_status()
File ".../aiohttp/client_reqrep.py", line 636, in raise_for_status
    raise ClientResponseError(

Environment

  • livekit-agents: 1.5.1
  • livekit-protocol: 1.1.3
  • livekit: 1.1.3
  • Python: 3.11
  • Deployment: Self-hosted agent connecting to LiveKit Cloud
  • Agent type: Voice agent (STT + LLM + TTS pipeline)

Steps to reproduce

  1. Enable agent recording (enable_recording: true in job dispatch, or pass record=True to session.start())
  2. Start a voice agent session
  3. End the session (participant disconnects)
  4. Observe the error in agent logs during session cleanup

What we’ve tried

  • Upgraded all LiveKit packages from 1.3.x to 1.5.1 — same error
  • Confirmed the JWT is generated with ObservabilityGrants(write=True)
  • Confirmed the request is well-formed (protobuf header + chat history JSON + audio)
  • The agent session itself works correctly — the error only occurs during the post-session observability upload

Notes

  • The 500 is a server-side error from the /observability/recordings/v0 endpoint — the client successfully constructs and sends the request
  • Agent observability is enabled in the LiveKit Cloud project settings (enable_recording: true appears in the job dispatch)
  • The error is non-fatal but produces noisy logs on every session
  • Is there any additional project-level configuration required to enable the observability backend?
1 Like
2

Do you have a session ID where this happened? It will start with RM_

3

Yes I have it right here: RM_FXVA4kgq7Gg3

1 Like
4

Thanks for reporting. We are looking into it.

1 Like
5

I am facing same issue with my agent too. @CWilson @darryncampbell

Please see the attached image of the logs.

Screenshot 2026-04-07 at 6.33.14 PM
Screenshot 2026-04-07 at 6.33.14 PM2402×390 77.6 KB

6

@Rajan_kumar Which session is this please? (starts with RM_), just want to double-check this is the same issue.

7

@Matin_Massoudi It looks like your project was affected by this incident: LiveKit Status - LiveKit Cloud Dashboard Missing Observability Sessions

@Rajan_kumar I am less sure that your issue is related to this incident, but it depends when your affected project was created.

8

Fixed! Thank you for reporting back