How can I improve STT accuracy with Deepgram Nova-3?

Getting Started
1

I’m currently using Deepgram Nova-3 for real-time speech-to-text, but I’m facing several transcription accuracy issues. I’m looking for suggestions on improving STT accuracy, recommended configuration changes, debugging techniques, or any best practices you follow. References, tools, or real-world experiences would be greatly appreciated.

2

What is your setting? Would you mind sharing a bit more information on your current setup, like is the speaker talking mostly over the phone, or maybe to a device a bit further away. Are the talking indoors, or mostly outside?

You have also plenty of options on the Deepgram side, one the most helpful may be playing around with endpointing.

3

Hi Pawel,

Before tuning anything, worth pinning down two things, since they change the answer a lot: what language, and web or telephony? If it’s English and the pain is turn boundaries (clipped/merged speech) rather than wrong words, the fix is different from a vocabulary problem. With that caveat, here’s where I’d look, cheapest first.

1. Flip the formatting defaults (one-line win). The LiveKit plugin ships with smart_format=False, numerals=False, and filler_words=True. If the complaint is “the transcript looks wrong” (odd casing, spelled-out numbers, ums everywhere), setting smart_format=True fixes a big chunk of that in a single line before you touch anything else.

2. keyterm, not keywords. If you’re missing names, jargon, or product vocab: the plugin exposes both params, and a lot of people carry keywords over from Nova-2, where it’s a no-op on Nova-3. Move them to keyterm. Nova-3 multi takes up to ~500 tokens (~100 words) and passing several is expected, so load in the vocab you actually care about.

3. Turn detection first, endpointing second. This is the usual culprit for clipped or merged words in an agent. In a LiveKit agent the turn boundary is normally owned by LiveKit’s turn detector, and the Deepgram plugin already defaults to endpointing_ms=25. If you start tuning Deepgram endpointing while LiveKit’s VAD is also segmenting, the two fight each other and the clipping gets worse. Decide who owns the boundary first, then tune only that path.

4. If it’s English and the real pain is turn boundaries, try Flux. Running Flux through the v2 plugin path with turn_detection="stt" moves end-of-turn detection into the STT model itself, instead of layering a separate VAD/turn detector on top that competes with it. For English voice agents where the words are fine but the turn-taking is off, that’s usually the actual answer, not endpointing knobs.

5. Audio path. If the words are genuinely wrong (not just formatted oddly), sanity-check sample rate and encoding, and whether AEC/noise suppression is helping or hurting. Telephony (8kHz) and far-field mics cost you more WER than the model choice does.

If you can share your request params (model, language, endpointing/turn-detection settings) and one expected-vs-actual transcript, I can point at the specific thing to change.

4

Hi Ed / Pawel — thanks for the pointers. Sharing our setup and where we’re stuck.

We support both telephony (SIP) and WebRTC, but most production traffic (and the accuracy pain) is telephony. Language is mostly English, with language="multi" for mixed callers.

Current config overview

Deepgram (via LiveKit plugin):

  • model=nova-3
  • language= multi/ single depend on agent
  • detect_language=False
  • interim_results=False
  • punctuate=False
  • smart_format=False
  • filler_words=True
  • numerals=True
  • endpointing_ms=300
  • sample_rate=16000
  • keyterm= boost list (numbers, months, weekdays — Nova-3 keyterm, not keywords)

LiveKit turn stack (we treat LiveKit as the turn owner, not Deepgram endpointing alone):

  • Silero VAD: min_speech_duration=0.03, min_silence_duration=0.55, prefix_padding_duration=0.3, activation_threshold=0.3, sample_rate=8000
  • Turn detector: LiveKit TurnDetector (v1-mini locally for benches)
  • Endpointing: mode=fixed, min_delay=0.5, max_delay=1.5
  • Adaptive interruption enabled
  • Noise cancellation: BVCTelephony for SIP, BVC for WebRTC

Pain is a mix of wrong words (names / menu / short answers) and turn boundaries (clipped last words, missed yes/no, merged turns) on phone audio.

I’ve tried flipping formatting (smart_format / numerals / fillers) and we’re already on keyterm. Still chasing whether the bigger lever is audio path (8k telephony + BVC) vs turn detection vs Nova-3 itself — and whether Flux + turn_detection="stt" is worth trying for English telephony.

Benchmarking ask

I’m also looking for a solid way to A/B configs against real call recordings.

We have user-only WAVs from production (Insights / egress). I’ve built / tried several scripts:

  • Deepgram-only file STT
  • Offline inject into AgentSession (VAD + STT + turn, no room)
  • Live mic worker
  • Full room replay (publish recording as a SIP-like mic track into a LiveKit room)

The problem: none of them really “benchmark.” With our prod config they all tend to look fine / give the “correct” transcript, so they don’t surface the failures we see live on telephony. I’m trying to get something that:

  1. Replays a recording through a path close to prod (esp. SIP + BVCTelephony)
  2. Lets me change VAD / endpointing / Deepgram params and quantify the difference (WER vs reference, missed short answers, EOU latency, empty finals, turn splits)
  3. Doesn’t just rubber-stamp the current prod settings

Is there a recommended LiveKit / Deepgram workflow for recording-based STT regression benches? Happy to share expected-vs-actual examples once I have a setup that actually differentiates configs.

5

Can you share a few RequestId’s so we can see if there’s something we can identify on our end that might help you? You can send them to ed.charbeneau@deepgram.com

Thanks.