Question: Built-in LiveKit way to strip silence before STT?

Getting Started
,
1

“Does LiveKit VAD prevent silent audio from being sent to STT providers?”

I was using the Livekit Deepgram STT plugin with LiveKit and noticed that a significant amount of silence audio was still being streamed to Deepgram, which was increasing my STT usage/costs (almost 2x in my case).

I already had VAD enabled, but it seemed like VAD was mainly helping with turn detection and was not fully preventing silent audio frames from reaching the STT provider. LiveKit’s VAD/turn detection handles speech activity and endpointing, but it does not necessarily act as a hard audio gate before every STT provider call. VAD and turn detection configuration guide | LiveKit

To solve this, I added a VAD gateway layer that sits between the audio input and Deepgram:

  • Run VAD on incoming audio frames
  • Drop/strip silence frames
  • Forward only speech segments to Deepgram

This reduced unnecessary STT processing and avoided paying for silence.

Solution gist:
https://gist.github.com/kaushal-aubie/19c853f568e464c8485efb7f25a9d8a6

Question:
Is this expected behaviour when using LiveKit + external STT providers like Deepgram, or is there a built-in LiveKit way to gate/filter audio before it reaches STT?

Would be interested to know if there is a recommended approach/configuration for this use case.

2

@Kaushal_Shah, Expected, and it’s two layers deep, not a config you’re missing. push_audio fans every input frame out to STT, VAD, AMD and the turn detector in parallel [ audio_recognition.py ]; VAD is a parallel consumer for endpointing, it never gates what STT sees. The Deepgram plugin’s send loop then forwards every frame it gets to the socket in 50ms chunks [ deepgram stt.py ]. Deepgram bills by audio submitted, and its KeepAlive is the documented way to hold the socket open through silence without charge (“KeepAlive messages alone don’t incur transcription charges”) [ Audio Keep Alive | Deepgram's Docs ]. So a mostly-silent session bills its full duration, which is your ~2x.

No built-in gate flag exists, so your gateway is the right idea; the framework-native spot for it is an stt_node override, which receives the AsyncIterable[rtc.AudioFrame] headed to STT [ agent.py ]. Run VAD there and push_frame only speech. The session’s own VAD and turn detector get their frames upstream of stt_node, so gating there cuts STT cost without touching endpointing or interruptions. Pad a couple hundred ms around each segment so Deepgram’s smart_format doesn’t degrade on hard cuts.

3

@Muhammad_Usman_Bashir
Thanks, i will try that out to see it improves the silence stripping more, and it seems more framework aligned solution, will let you know how it goes.

EDIT: i have 3 agents so overriding stt node for each agent was more code duplicaation, and instead i am passing stt instance which has vad gateway attatched which was more decoupled and easy solution

4

Also i think should livekit have something built in option or feature to have it, as it will really benefit the users. The cost was not minimal but around 40% which was huge for me. So if team develop something more properly to pass as settings, would it be great?

Shall i raise as feature request?

5

No reason not to ask, but a few reasons why it might not be taken up (off the top of my head)

  • Like you say, I believe Deepgram do charge by audio processed, but I don’t think that is ubiquitous, and this technique wouldn’t help with STT providers who charge by connection time.
  • There would be a danger of clipping actual audio when detecting silence, I would imagine. Similarly if the audio levels are low these might be misclassified (or it could lead to more error-prone configuration).
  • This would probably break the STT provider’s turn detection, so it makes the configuration more complex (need to disable this if turn_detection=stt)
  • I’m not sure how it would interact with interrupt detection, or whether it would slow it down.
6

I agree there would be lot of things to consider, hmm, i think we can always extend stt node or use vad gateway for STT, so i think that is already good in terms of livekit customization.

I think yeah we can leave it as is, and someone who wants to save cost can use above gist or extend stt node with their own implementation to stripe silence if needed