LiveKit + Deepgram STT not understanding Hindi voice (real-time issue)

Getting Started
1

I am implementing a real-time voice assistant using LiveKit Agents with Deepgram STT and ElevenLabs TTS.

Problem:
Deepgram is not correctly recognizing Hindi speech in live streaming. English works fine, but Hindi or Hinglish is either गलत transcribed or not detected.

Current setup:

  • STT: Deepgram (nova-2 / nova-3)

  • Language: hi / multi

  • LiveKit AgentSession

  • VAD: silero

  • Noise cancellation enabled

Issues:

  • Hindi sentences are cut or misinterpreted

  • Hinglish fails completely

  • Latency is also high

Example:
Input: “नमस्ते आप कैसे हैं”
Output: incorrect or empty

What I tried:

  • language=“hi”

  • language=“multi”

  • changing endpointing delay

  • using noise cancellation

Questions:

  1. Which Deepgram model works best for Hindi in streaming?

  2. Should I use “multi” or “hi”?

  3. Is LiveKit inference STT better or direct Deepgram plugin?

  4. Any best practices for low-latency Hindi voice?

Goal:
Achieve real-time (<500ms) accurate Hindi conversation.

Any help would be appreciated.

2

That’s a very interesting case and I keep hearing more and more about it. My guess would be that it’s about STT model. Have you tried different ones, like Soniox or ElvenLabs Scribe?