Openai TTS sends stream_format="sse" to every non-OpenAI backend, so compatible endpoints return audio that gets thrown away

Agents
,
1

Hi team,

I think there’s a bug in the OpenAI TTS plugin that breaks any OpenAI-compatible backend. Two people have hit it on the forum without anyone finding the cause, so I dug into it.

The plugin decides how to read the response based on the model name rather than what the server actually sends back. In tts.py:

  AUDIO_STREAM_MODELS = {"tts-1", "tts-1-hd"}

  def synthesize(self, text, *, conn_options=DEFAULT_API_CONNECT_OPTIONS):
      if self._opts.model in AUDIO_STREAM_MODELS:
          return AudioChunkedStream(tts=self, input_text=text, conn_options=conn_options)
      return SSEChunkedStream(tts=self, input_text=text, conn_options=conn_options)

Anything that isn’t tts-1 or tts-1-hd goes to SSEChunkedStream, which sends stream_format=“sse” and only reads lines starting with data: . But stream_format is specific to OpenAI. Other servers just ignore the unknown field and return normal MP3 or WAV bytes, so no line ever matches and push() never gets called.

From there pushed_duration() is 0, so the emitter raises no audio frames were pushed for text: … Since APIError defaults to retryable=True and max_retry is 3, the provider gets called four times for a single sentence. All four succeed and get billed, and then the turn fails anyway. The HTTP response is a clean 200 the whole time, which is why it looks like a provider problem.

Worth noting that DEFAULT_MODEL is gpt-4o-mini-tts, so the default also lands on the SSE path. Right now the only way to reach the working branch is to name the model tts-1 or tts-1-hd.

To reproduce:

  from livekit.plugins.openai import TTS

  tts = TTS(
      base_url="https://api.deepinfra.com/v1",
      api_key="<key>",
      model="hexgrad/Kokoro-82M",
      voice="af_bella",
  )

Any OpenAI-compatible server does the same thing (DeepInfra, Kokoro-FastAPI, vLLM, LM Studio, Speaches). A plain curl to the same endpoint returns valid
audio.

Reported before, cause never found:

Confirmed on the current release, 1.6.9.

Would it make sense to pick the parser from the response Content-Type instead of the model name? Or simply expose stream_format as a constructor argument, so people using other backends can pass stream_format=“audio”. Either would fix it. Happy to open a PR if you’d like.

For anyone stuck in the meantime:


  class CompatTTS(TTS):
      def synthesize(self, text, *, conn_options=DEFAULT_API_CONNECT_OPTIONS):
          return AudioChunkedStream(tts=self, input_text=text, conn_options=conn_options)
2

Hi @abidullahcs.uk , nice work.

I was able to reproduce this issue and verify your workaround. I think the next best step would be to raise a PR for the engineering team to weigh in on your fix, so if you could raise that, it would be appreciated. Personally I would lean towards picking the parser from the content type.

Claude also suggested another workaround which seems to work in my testing:

from livekit.plugins.openai.tts import AUDIO_STREAM_MODELS
AUDIO_STREAM_MODELS.add("hexgrad/Kokoro-82M")   # do this once, at import time
3

Thanks @darryncampbell, and thanks for taking the time to reproduce it. I’ll get the PR up.

Content type works, I checked. AsyncStreamedBinaryAPIResponse inherits headers from BaseAPIResponse, so you can read stream.headers[“content-type”] inside the async with before touching the body, then use iter_lines() for text/event-stream and iter_bytes() for everything else. Both stream classes collapse into one, which reads better than the split we have now.

One question before I write it. stream_format is an OpenAI-only parameter. DeepInfra just ignores it, but a stricter server could reject the request outright, so I was planning to send it only for the known OpenAI models and let the content type handle the rest. Let me know if you’d rather keep the request shape the same for everyone.

Also your AUDIO_STREAM_MODELS.add(...) is much tidier than my subclass. Only thing worth noting for anyone copying it is that the set is module level, so it applies to every TTS instance in the process.

4

@darryncampbell PR is up: fix(openai): read TTS response by content-type, not model name by abidullahdev · Pull Request #6930 · livekit/agents · GitHub

One change from what I said above. I’d planned to send stream_format only for known OpenAI models, but dropped it gpt-4o-mini-tts-2025-12-15 isn’t in either allowlist, and omitting the field routes it to the raw-audio branch, which is the only one that doesn’t record token usage. Usage metrics would quietly go to zero for it and every future snapshot.

So the PR only changes how the response is read, never what’s sent. Existing OpenAI requests are byte-identical. Happy to add the request-side change separately if you’d still like it.

5

Looks like it’s already been approved :lk-launch:

If you can please address that final comment from Long, then merge and it will be included in the next release.