LiveKit Approach for Live Captions & Meeting Transcripts

Getting Started
1

Question: Recommended LiveKit Approach for Live Captions & Meeting Transcripts

We currently have a custom transcription setup in our LiveKit-based video conferencing application and would like to know the recommended production-grade approach supported by LiveKit.

Current Setup

  • Live captions are generated client-side using:
    • Web Speech API (Chrome/Edge)
    • Whisper running in browser workers (Firefox/Safari)
  • Captions are distributed using LiveKit Data Channels.
  • Meeting transcripts are collected, merged, and stored through our own backend.
  • We also maintain a separate Python Whisper service that receives audio streams over WebSockets for server-side transcription.

Challenges

Maintaining multiple transcription paths (Web Speech, browser Whisper, Python Whisper) has become complex and resource-intensive. We’re handling our own VAD, silence detection, filtering, deduplication, and transcript storage.

Questions

  1. Is the recommended LiveKit approach today to use LiveKit Agents + a speech provider (Deepgram, AssemblyAI, OpenAI, Whisper, etc.) and consume transcripts via RoomEvent.TranscriptionReceived?
  2. Does LiveKit provide any built-in transcription/captions solution that can be enabled without running our own agents?
  3. Are there official examples or reference architectures for implementing live captions and persisted meeting transcripts in a Next.js application?
  4. For teams already running transcription in production with LiveKit, what architecture would you recommend instead of a custom client-side STT implementation?

We’d appreciate any guidance on the most maintainable and scalable solution supported by the LiveKit ecosystem.

2

Hi Sahiti, welcome! Great questions — you’ve basically described the exact situation LiveKit Agents was built to simplify.

Taking them in order:

1. Is Agents + a speech provider the recommended approach?

Yes, with one update: transcriptions are now delivered over text streams on the lk.transcription topic rather than RoomEvent.TranscriptionReceived — that event (and publish_transcription()) is deprecated. On the client you’d consume captions like this:

room.registerTextStreamHandler('lk.transcription', async (reader, participantInfo) => {
  const message = await reader.readAll();
  // participantInfo.identity is the speaker; reader.info.attributes['lk.transcribed_track_id'] ties it to a track
});

Or in React, just use the useTranscriptions hook. Full details in the Text and transcriptions docs.

2. Is there a built-in solution without running your own agent?

There’s no “flip a switch on the room” transcription feature — you do need an agent in the room to do the STT. But it’s a small amount of code, and two things reduce the operational burden a lot:

  • LiveKit Inference gives you STT (Deepgram, AssemblyAI, Cartesia, etc.) through your LiveKit API key — no separate provider accounts or API keys to manage.
  • If you’re on LiveKit Cloud, you can deploy the agent as a Cloud Agent, so there’s no infrastructure for you to run either.

3. Official examples for captions + persisted transcripts in Next.js?

  • multi-user-transcriber.py — the key example for your use case: one agent that spins up an AgentSession per remote participant and transcribes everyone in the room.
  • live-translated-captioning — full-stack example with an agent transcribing (and translating) speech into live captions rendered by a TypeScript web client.
  • transcription-frontend — a caption UI you can crib from for your Next.js app.

4. Recommended architecture to replace client-side STT

A single server-side transcription agent per room, replacing all three of your current paths (Web Speech, browser Whisper, and the WebSocket Whisper service):

  1. Agent joins the room via dispatch and creates an AgentSession per participant (as in the multi-user-transcriber example), with an STT plugin of your choice.
  2. VAD, endpointing/turn detection, and interim vs. final results are handled by the framework — all the pieces you’re currently hand-rolling (silence detection, filtering, dedup) go away.
  3. Live captions: delivered automatically to all clients over lk.transcription text streams, attributed to the correct speaker. Clients just render; no client-side STT at all.
  4. Persisted transcripts: in the same agent, accumulate finalized turns (e.g. in on_user_turn_completed or from the session history) and write the merged transcript to your backend/storage in a shutdown callback when the room ends. Since one process sees every participant’s finalized text with speaker identity and timing, the merge/dedup step largely disappears too.

This gets you one code path, consistent caption quality across all browsers, and speaker-attributed transcripts for free. Happy to go deeper on any of these!

3

Thanks — this is exactly what we needed. We’re on a self-hosted LiveKit server, so a few follow-ups before we commit:

  1. LiveKit Inference on self-hosted? Is Inference available for self-hosted servers, or does that require LiveKit Cloud / a LiveKit account? If self-hosted, we’d use our own Deepgram/AssemblyAI/OpenAI API keys directly via the agent plugin — just want to confirm Inference isn’t a prerequisite.
  2. Agent hosting/ops on self-hosted. We know Cloud Agent isn’t an option — we’d run the agent process ourselves. Any recommended deployment pattern? One long-lived agent process that accepts dispatch for any room (worker pool), or one agent per room? And is there a documented token/API setup for a self-hosted agent joining rooms (access token + server URL + a dedicated participant name)?
  3. Dispatch wiring on self-hosted. Do we configure an agent-dispatch webhook ourselves on the self-hosted server, and does the standard RoomServiceClient / createDispatchToken flow in livekit-server-sdk@2.14.2 work as-is? Also — is there a minimum server version required for lk.transcription text streams to reach clients (since publishTranscription is deprecated)?
  4. Scale of the multi-user pattern. The multi-user-transcriber.py example spins one AgentSession per remote participant. With 10–20 participants that’s N concurrent STT streams and the agent subscribing to all audio tracks — fine on a self-hosted agent, or is there a ceiling where a single agent per room starts to struggle? Any way to share one STT session with diarization instead?
  5. Persisted transcripts. You suggested accumulating finalized turns and writing them to our backend in a shutdown callback. Our stack already has a Next.js /api/save-transcript endpoint (Redis + JSON storage). Is “agent HTTP-POSTs merged segments back to the Next.js API” the standard pattern, or does the Agents framework have any built-in persistence we should use instead?
  6. Agent restarts / missed audio. If the agent dies mid-meeting and dispatch respawns it, do clients re-see earlier caption text (replay from lk.transcription history), or is the transcript lost from that gap?
    Thanks again
4

Hi Sahiti,

LiveKit Inference on self-hosted? Is Inference available for self-hosted servers, or does that require LiveKit Cloud / a LiveKit account? If self-hosted, we’d use our own Deepgram/AssemblyAI/OpenAI API keys directly via the agent plugin — just want to confirm Inference isn’t a prerequisite.

LiveKit Inference is not available for self-hosted LiveKit servers and you would need to use your own API keys directly via the agent plugins. The nuance comes with customers who choose to use LiveKit cloud and self-host their own agents, and in that scenario LiveKit Inference IS available, however you are not in that category.

Agent hosting/ops on self-hosted. We know Cloud Agent isn’t an option — we’d run the agent process ourselves. Any recommended deployment pattern? One long-lived agent process that accepts dispatch for any room (worker pool), or one agent per room? And is there a documented token/API setup for a self-hosted agent joining rooms (access token + server URL + a dedicated participant name)?

The documentation for self-hosted agent deployments is here: Self-hosted deployments | LiveKit Documentation and is the same regardless of whether you are using LiveKit cloud for media, or self-hosting LiveKit. Just avoid burstable instance types.

Dispatch wiring on self-hosted. Do we configure an agent-dispatch webhook ourselves on the self-hosted server, and does the standard RoomServiceClient / createDispatchToken flow in livekit-server-sdk@2.14.2 work as-is? Also — is there a minimum server version required for lk.transcription text streams to reach clients (since publishTranscription is deprecated)?

The deployment logic along with token flow is the same for both LiveKit Cloud and self-hosted since that’s all open source. Agent dispatch | LiveKit Documentation

I’m not sure what the minimum server version is for lk.transcription, but it will work with the latest release (or, it should do).

Scale of the multi-user pattern. The multi-user-transcriber.py example spins one AgentSession per remote participant. With 10–20 participants that’s N concurrent STT streams and the agent subscribing to all audio tracks — fine on a self-hosted agent, or is there a ceiling where a single agent per room starts to struggle? Any way to share one STT session with diarization instead?

We have software we use to host meetings internally with agent transcription that can handle that many participants (and more)

That’s a good question about diarization, I suspect the accuracy of transcriptions for that many participants would struggle, so I would recommend the one user per track approach.

Our docs AI hallucinated the following, but I was happy to see he agreed with me :slight_smile: “One important caveat: accuracy generally drops as simultaneous speakers increase, especially with overlap/crosstalk. The docs describe primary-speaker detection as RMS/loudness based, so in crowded conversations the “active speaker” can shift rapidly. For meetings with many participants, separate audio tracks per participant usually produce better results than diarizing a single mixed room feed.”

Persisted transcripts. You suggested accumulating finalized turns and writing them to our backend in a shutdown callback. Our stack already has a Next.js /api/save-transcript endpoint (Redis + JSON storage). Is “agent HTTP-POSTs merged segments back to the Next.js API” the standard pattern, or does the Agents framework have any built-in persistence we should use instead?

Nothing built in for this. In general it is best to use the on_session_end callback for tidy up

Agent restarts / missed audio. If the agent dies mid-meeting and dispatch respawns it, do clients re-see earlier caption text (replay from lk.transcription history), or is the transcript lost from that gap?

The best resource for agent behaviour after restart is this blog from CWilson: Why Your Agent Leaves on Browser Refresh (and When to Keep It) | LiveKit