Inaccurate Transcript Despite Correct Realtime Understanding

royibernthal · August 8, 2026, 10:04pm

I’m using the following AgentSessionOptions for realtime speech in Hebrew:

{
  llm: new openai.realtime.RealtimeModel({
    model: 'gpt-realtime-2.1',
    voice: 'marin',
    inputAudioTranscription: {
      model: 'gpt-4o-mini-transcribe',
      language: 'he',
    },
  }),
}

The responses I’m getting from the model during the conversation suggest that it understands what I’m saying. However, when I look at the transcript in the session record, many of the things I said appear to be only partially understood, misunderstood entirely, or just plain gibberish.

This leads me to believe that the recorded transcript is different from the speech transcript the realtime model interprets and uses during the conversation.

How can I make the recorded transcript contain the same interpreted speech transcript used by the realtime model? Alternatively, how can I record a more faithful transcript?

darryncampbell · August 8, 2026, 10:32pm

Hi Royi, this isn’t possible because the model is reasoning on the audio, it isn’t converting it to a transcript first.

You can specify a separate STT() to generate a transcript, like this:

session = AgentSession(
    stt=deepgram.STT(),
    # OpenAI Realtime API
    llm=openai.realtime.RealtimeModel(
        voice="alloy",
        input_audio_transcription=None,  # use Deepgram STT instead
    ),
)

The realtime model will still reason on the audio, as before, but the transcripts you receive will be from the dedicated STT which should be more accurate. You can swap the STT out for whichever provider works best for Hebrew (based on your language he) - I would be interested in your opinions on that.

royibernthal · August 8, 2026, 11:21pm

Hi Darryn, you’re right it’s STS.

A few questions to make sure I understand what’s going on:

Doesn’t assigning STT in the AgentSession in addition to the realtime model add another hop and additional latency to each conversation turn? Does the realtime model ignore it entirely? Does it simply run in parallel with the realtime model and get used for transcription without blocking it?

Is the inputAudioTranscription I originally passed to the realtime model basically used for transcription separately from the realtime model itself? Is that equivalent to setting AgentSessionOptions.stt to gpt-4o-mini-transcribe?

Also, probably need to resolve this issue before I can give Hebrew STT a proper try:

Pawel_Lach · August 9, 2026, 7:15am

As far as I understand setting up the transcription model is a completely separate process, that does not have any effect on the real-time model at all. It’s rather some sort of a debugging tool, to “guess” what goes into the real-time model. Nevertheless, also curious how this blackbox works.

CWilson · August 10, 2026, 4:15pm

No, it does not add another hop or addtional latency.

When you set both a realtime model and stt on AgentSession, the agent forks the audio. Every incoming audio frame is pushed to two places at once: the realtime session and the STT stream (see push_audio() in agent_activity.py — it calls self._rt_session.push_audio(frame) and self._audio_recognition._push_audio(frame) back to back).

flowchart LR
    MIC[User audio] --> RT[Realtime model<br/>reasons on raw audio]
    MIC --> STT[STT provider<br/>e.g. Deepgram]
    RT --> RESP[Agent response]
    STT --> TX[user_input_transcribed events<br/>+ session history/transcript]

royibernthal · August 13, 2026, 12:38pm

Makes sense, thanks for the explanation.

Is AgentSessionOptions.stt the same as RealTimeModelOptions.inputAudioTranscription except not limited to the real time model provider’s STT models?

darryncampbell · August 14, 2026, 12:27am

Yes, but I would usually recommend a dedicated STT as it gives you more control and additional features, such as keyterms

royibernthal · August 14, 2026, 5:53pm

Good to know. Is there any way to pass keyterms to realtime models as well?

Also, since your team just enabled Hebrew for Deepgram, I was able to test the Deepgram STT after updating all @livekit packages to version 1.6.3:

{
stt: new inference.STT({ model: 'deepgram/nova-3', language: 'he' }),
llm: new openai.realtime.RealtimeModel({
model: 'gpt-realtime-2.1',
voice: 'marin',
}),
};

I’m getting the following warning and error while the agent is still saying their first message, before I event get to speak. To rule out that it’s happening directly because of the packages update - it’s worth noting that I’m only getting it for the new config, while the previous config still runs with no errors.

WARN (27080): failed to recognize speech, retrying in 2000ms
stt: “inference.STT”
attempt: 2
error: {
“type”: “APIConnectionError”,
“message”: “Error connecting to LiveKit WebSocket”,
“stack”:
APIConnectionError: Error connecting to LiveKit WebSocket
at WebSocket.onError (C:\Projects\myproject\livekit-agent\node_modules.pnpm@livekit+agents@1.6.3_@live_205979579f038fa20adc56afc8d178ed\node_modules@livekit\agents\src\inference\utils.ts:122:16)
at Object.onceWrapper (node:events:623:26)
at WebSocket.emit (node:events:508:28)
at emitErrorAndClose (C:\Projects\myproject\livekit-agent\node_modules.pnpm\ws@8.21.0\node_modules\ws\lib\websocket.js:1060:13)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)
“body”: null,
“retryable”: true,
“name”: “APIConnectionError”
}

ERROR (16128): AgentSession is closing due to an unrecoverable error
type: “stt_error”
label: “inference.STT”
recoverable: false
error: {
“type”: “APIConnectionError”,
“message”: “Error connecting to LiveKit WebSocket”,
“stack”:
APIConnectionError: Error connecting to LiveKit WebSocket
at WebSocket.onError (C:\Projects\myproject\livekit-agent\node_modules.pnpm@livekit+agents@1.6.3_@live_205979579f038fa20adc56afc8d178ed\node_modules@livekit\agents\src\inference\utils.ts:122:16)
at Object.onceWrapper (node:events:623:26)
at WebSocket.emit (node:events:508:28)
at emitErrorAndClose (C:\Projects\myproject\livekit-agent\node_modules.pnpm\ws@8.21.0\node_modules\ws\lib\websocket.js:1060:13)
at process.processTicksAndRejections (node:internal/process/task_queues:90:21)
“body”: null,
“retryable”: true,
“name”: “APIConnectionError”
}

darryncampbell · August 17, 2026, 8:04am

Looking at the server logs, I believe you have run out of Inference STT credits.

royibernthal · August 18, 2026, 12:14pm

Where can I see how many inference STT credits I have left?

I still got a transcript when using realtime models in English. Would that still make sense if I’ve run out?

darryncampbell · August 18, 2026, 12:54pm

Where can I see how many inference STT credits I have left?

There is no realtime dashboard, but you should have received an email saying you were close. Related, looking at your project billing page it looks like you have exceeded the inbound local minutes.

royibernthal · August 18, 2026, 3:14pm

“looking at your project billing page, it looks like you have exceeded the inbound local minutes.”

What indicator are you looking at?

I received an email on July 29 saying that I had used 80% of my free inference credits. Definitely no such email for August.

I’ve tested it again now:

Inbound works fine, so I’m not sure how to reconcile that with having run out of inbound local minutes.

STT now works without an error. Perhaps the inference STT credits have been reset since the last time I tried it, despite it being mid-month?

Does upgrading from the free plan remove the inference credits cap entirely?

Will setting my own DEEPGRAM_API_KEY in the env vars remove the dependency on LiveKit’s inference credits?

That said, the STT transcript is sometimes gibberish, sometimes in Latin characters, and sometimes a semi-accurate Hebrew transcription, despite the real-time model understanding what I’m saying.

It’s worth noting that testing the same STT on Deepgram’s site consistently yields near-perfect results in Hebrew, so I’m not sure why it’s different when using LiveKit.

I shared the session with you via PM.

darryncampbell · August 19, 2026, 9:38am

I was looking here: https://cloud.livekit.io/projects/p_/settings/billing

I’m a bit confused by this, but if it is working now then I won’t argue. Upgrading to ship gives you additional credits and once you hit the free allowance, it won’t cap you:

Yes, but you’ll also have to update your code to point at the Deepgram plugin rather than use LiveKit inference.

I ran some tests and I figured out what is going on.
Although you correctly configured the separate STT with the realtime LLM, you are still seeing the transcript from the realtime LLM in the agent observability, that is why you are seeing the keyboard icon next to the user transcripts there.

Firstly, set inputAudioTranscription to null in your RealTime Model

    llm: new openai.realtime.RealtimeModel({
      model: 'gpt-realtime-2.1',
      voice: 'marin',
      inputAudioTranscription: null,
    }),

Then, register to receive user_input_transcribed, that is where the STT events will be received:

These will come from Deepgram, so I would expect the accuracy to match what you see on the Deepgram site.

Note that the transcript in observability is driven by the conversation handler, so will now be empty for user turns.

royibernthal · August 19, 2026, 8:09pm

Thanks for the detailed response. Appreciate your patience with all the issues I’m running into.

Billing

I see this in my billing page regarding local inbound minutes:

Build: 50 incl. local inbound minutes
$0.00
Build: Additional local inbound minutes ($0.01 per minute)
$1.27

Doesn’t it mean I passed the capped free minutes and I’m being charged for overage (despite being on the free plan)? If so, maybe it’d explain why it’s still working.

Listening to user_input_transcribed

Setting inputAudioTranscription = null and listening to user_input_transcribed yields correct transcripts.

User transcripts in observability are empty now though as you said. Any way for me to configure it in a way that would be connected to the conversation handler and appear in observability and session report? Or is there a bug that would have to be fixed on your end for that to happen?

Deepgram API key

I switched to the Deepgram plugin and specified a DEEPGRAM_API_KEY in the env, for some reason it can’t connect - forbidden which is weird.

failed to connect to Deepgram, retrying in 10 seconds: Error: Unexpected server response: 403 (3/32)

Logging process.env.DEEPGRAM_API_KEY shows the correct API key, and that Deepgram project has more than sufficient credits.

No logs appear on Deepgram’s side.

How would you go about debugging this?

Logger

I might be wrong, but I suspect LiveKit’s logger (e.g. logger.info) messes up Hebrew text encoding, it prints mojibake.

When using console.log Hebrew characters are printed correctly, so the issue doesn’t seem to be with PowerShell’s decoding.

darryncampbell · August 20, 2026, 9:21am

That is a mystery to me, I expected it to be capped since, without a credit card, we have no way of reclaiming that $1.27

I see an internal discussion from last year that this is a known issue, but I don’t see any documentation to this effect. I’ll follow up to either get this documented, or revisited, but I’m not optimistic this will be changed in the short term I’m afraid.

If this were me, I’d assume that the incorrect key was being sent to Deepgram for some reason, or the wrong key. I would start from scratch with a new agent if it were me, clearing out my machine environment variables just to be sure. If testing with a hosted agent, make sure the secret exists under your agent configuration on the dashboard

Is there a way I can easily test and verify this, as a non-Hebrew speaker? Where are you seeing the incorrect output?

royibernthal · August 20, 2026, 2:35pm

That is a mystery to me, I expected it to be capped since, without a credit card, we have no way of reclaiming that $1.27

True. Bug?

I see an internal discussion from last year that this is a known issue, but I don’t see any documentation to this effect. I’ll follow up to either get this documented, or revisited, but I’m not optimistic this will be changed in the short term I’m afraid.

Thanks. Would be great if you could push this. Seems very necessary for realtime in less popular languages.

If this were me, I’d assume that the incorrect key was being sent to Deepgram for some reason, or the wrong key. I would start from scratch with a new agent if it were me, clearing out my machine environment variables just to be sure. If testing with a hosted agent, make sure the secret exists under your agent configuration on the dashboard

I’m testing dev locally. Made sure the API key is fine by getting a proper response from a Deepgram curl.

It’s worth noting that this is forbidden rather than unauthorized, so perhaps the api key isn’t the culprit.

Is the deepgram plugin working for you?

Is there any way to print Deepgram’s “unexpected server response”? Perhaps some debug flag in the plugin?

Is there a way I can easily test and verify this, as a non-Hebrew speaker? Where are you seeing the incorrect output?

I’m seeing the incorrect output across the board - terminal, livekit cloud transcript tab, datadog, and the session report I’m parsing and storing on my own backend.

I would run the following Hebrew agent session options with the audio file from the shared session.

{

stt: new deepgram.STT({ model: ‘deepgram/nova-3’, language: ‘he’ }),

llm: new openai.realtime.RealtimeModel({

model: ‘gpt-realtime-2.1’,

voice: ‘marin’,

inputAudioTranscription: null,

}),

}

And compare logger.info vs console.log with Deepgram’s transcript:

session.on(voice.AgentSessionEventTypes.UserInputTranscribed, (event) => {

  logger.info(\`\[logger.info\] User input transcribed: ${event.transcript}, language: ${event.language}, final: ${event.isFinal}, speaker id: ${event.speakerId}\`);

  console.log(\`\[console.log\] User input transcribed: ${event.transcript}, language: ${event.language}, final: ${event.isFinal}, speaker id: ${event.speakerId}\`);

});

If you see any characters that don’t appear in the Hebrew alphabet, it’s a problem.

I know you’re not familiar with it, but it looks very different from mojibake and latin characters. Here it is:

א ב ג ד ה ו ז ח ט י כ ל מ נ ס ע פ צ ק ר ש ת

You can also just paste the transcript into ChatGPT / Claude / some other AI and ask if it’s proper Hebrew.

darryncampbell · August 20, 2026, 3:22pm

I need to look into this more thoroughly tomorrow, but just running a quick test I’ll paste my file below that is working for me with my Deepgram API key (I know it doesn’t match your setup with realtime, but I just wanted to verify it worked at all first).

deepgram-plugin.py (7.8 KB)

royibernthal · August 20, 2026, 6:21pm

I’ve tested deepgram tts with new agents using a pipeline instead of realtime - both with the node sdk and the python sdk.

It works fine with the python sdk, but I’m getting the same error with the node sdk. It seems to me there’s a bug with the deepgram node plugin.

Correct me if I’m wrong, but when using realtime, isn’t turn detection managed by the realtime model? Or does the turn detector still play a role? If it’s the former, then it’d likely only be relevant to this new isolated test and not to my original project.

Just to make sure we’re on the same page - in my original project I’m using the node sdk with a realtime model.

darryncampbell · August 21, 2026, 10:17am

I took a closer look at this, this morning. Sorry my last message was rushed & probably just added to the confusion.

I’ll attach the node agent I was testing with here

Deepgram STT works for me through plugin, you should be able to define an API key in .env.local (absent from the zip file for obvious reasons, but you can use the format of .env.example) and it will be picked up. One thing that usually trips me up is when I have keys in my environment outside of the app. If your key doesn’t work with the attached app, perhaps there is an issue with your key.
Nova-3 seems to be a good choice for Hebrew
Yes, the recommended turn detector to use is the realtime one. You can configure the session to use LiveKit’s turn detector, or the realtime one; you can also configure the realtime turn detector to use server_vad or semantic_vad. Although I was going to recommend server_vad, I can’t find any advice online but Claude seems to think semantic_vad would work best, which is the default.
I let Claude update the instructions to be more suited to a Hebrew conversation. The original instructions are from the agent starter.
I did try and test the console output but when I pasted the output into Claude it said it was sensible. I was holding up a YouTube video of somebody speaking Hebrew however, so my test was probably not ideal.

agent-starter-node-hebrew.zip (78.8 KB)

royibernthal · August 21, 2026, 9:42pm

I don’t have any relevant env keys outside the app, and inside the app it prints the correct ones defined in .env.local.

Deepgram works for me in your test project, yet it still doesn’t work in mine, so I’m a bit confused.

Do you mind taking a look? Finding that difference might give us a clue as to why it’s not working in the main project either.

livekit-test.zip (58.5 KB)

The difference between the loggers can be seen in your test project.

Out of curiosity, what’s the difference between what’s configured in createAgent (AgentCreateOptions) versus AgentSession (AgentSessionOptions)? Does the actual agent creation use a merged version of those configs? I imagine the former overrides the latter?

Topic		Replies	Views
Realtime model with Azure whisper STT Agents python , stt , realtime , openai , azure	17	349	February 26, 2026
Gpt realtime transcription misses Getting Started	1	43	June 18, 2026
Cannot get Deepgram STT working — server native or bot approach Self Hosting stt	7	49	July 27, 2026
Support for Live STT Partial Transcripts in Python SDK for OpenAI models Agents stt , openai	6	54	June 23, 2026
Realtime model is not working properly Agents realtime	8	127	May 30, 2026

Inaccurate Transcript Despite Correct Realtime Understanding

Related topics