Is voice.AgentSession intended for single-participant conversational agents?
Yes. voice.AgentSession is architected specifically for 1-on-1 agent-to-user conversational loops (an end-user interacting with an LLM bot).
Under the hood, AgentSession assigns a single “linked participant” (roomIO.linkedParticipant) to manage turn-taking, VAD, STT, and LLM state. Even though onTrackSubscribed fires for multiple participants when they publish audio:
-
AgentSession internally routes VAD triggers to the active session state.
-
The internal STT stream pipeline locks onto the first active user or expects a single stream for turn-handling.
-
Audio from Participant B arrives at the VAD, but because AgentSession’s internal turn state is either locked or expecting frames from the linked participant, it drops/skips EOU (End Of Utterance) for Participant B, producing audioTranscript: "".
Recommended Production Approach for Multi-Participant Meeting Transcription
To generate transcripts for every participant in a meeting room, you should not use voice.AgentSession. AgentSession includes unnecessary turn-handling, LLM orchestration, and single-user audio state tracking.
Instead, create a Transcriber Worker that connects directly to the LiveKit Room via ctx.connect(), listens to TrackSubscribed events for all participants, and attaches an independent STT stream (stt.stream()) to each audio track.
Official Node.js Pattern for Multi-Participant Transcription
Here is the clean production implementation in TypeScript / Node.js using @livekit/agents:
TypeScript
import {
type JobContext,
ServerOptions,
cli,
defineAgent,
} from '@livekit/agents';
import * as deepgram from '@livekit/agents-plugin-deepgram';
import { RemoteAudioTrack, Track, RoomEvent } from '@livekit/rtc-node';
import { fileURLToPath } from 'node:url';
export default defineAgent({
entry: async (ctx: JobContext) => {
// 1. Connect worker to room without AgentSession
await ctx.connect();
// 2. Initialize Deepgram STT instance
const stt = new deepgram.STT({
model: 'nova-3',
// apiKeys handled automatically by DEEPGRAM_API_KEY env var
});
console.log(`[Transcriber] Joined room: ${ctx.room.name}`);
// Helper function: Attach an independent STT stream per track
const handleAudioTrack = async (
track: RemoteAudioTrack,
participantIdentity: string
) => {
console.log(`[Transcriber] Starting STT stream for: ${participantIdentity}`);
// Create an independent STT stream for this participant
const sttStream = stt.stream();
// Forward PCM audio frames from LiveKit track into Deepgram stream
const audioStream = new track.AudioStream(track);
const audioTask = (async () => {
for await (const frame of audioStream) {
sttStream.push(frame);
}
})();
// Read transcription events coming back from Deepgram
const transcriptTask = (async () => {
for await (const event of sttStream) {
if (event.type === 'final_transcript' && event.text.trim()) {
console.log(`[Transcript] ${participantIdentity}: ${event.text}`);
// Option A: Publish transcript as a native text stream / data message back to room
const payload = new TextEncoder().encode(
JSON.stringify({
speaker: participantIdentity,
text: event.text,
timestamp: Date.now(),
})
);
await ctx.room.localParticipant.publishData(payload, {
topic: 'transcript',
reliable: true,
});
}
}
})();
// Clean up when track ends or participant leaves
track.once('ended', () => {
sttStream.close();
});
};
// 3. Subscribe to existing audio tracks
for (const participant of ctx.room.remoteParticipants.values()) {
for (const publication of participant.trackPublications.values()) {
if (
publication.track &&
publication.kind === Track.Kind.KindAudio &&
publication.track instanceof RemoteAudioTrack
) {
handleAudioTrack(publication.track, participant.identity);
}
}
}
// 4. Subscribe to new audio tracks as participants join/speak
ctx.room.on(
RoomEvent.TrackSubscribed,
(track, publication, participant) => {
if (
track.kind === Track.Kind.KindAudio &&
track instanceof RemoteAudioTrack
) {
handleAudioTrack(track, participant.identity);
}
}
);
},
});
cli.runApp(new ServerOptions({ agent: fileURLToPath(import.meta.url) }));
Why this fixes the issue:
-
Isolated Streams: Every participant gets their own stt.stream() instance connected to Deepgram. Participant A speaking into Stream 1 has zero impact on Participant B speaking into Stream 2.
-
No Turn-Locking: Eliminates voice.AgentSession VAD bottlenecks, preventing skipping EOU detection or empty audioTranscript: "" errors.
-
Infinite Scaling: Works cleanly for 2, 10, or 50 meeting participants simultaneously, bounded only by Deepgram API concurrency limits and host network bandwidth.