Openai TTS hardcodes response_format: 'pcm' and never checks the response, so a compatible endpoint returning anything else is played as raw samples

Agents
, , ,
1

Hi team,

Following on from livekit/agents#6930, which fixed a related problem in the Python plugin, I went looking for the same pattern here. The Node plugin has a different but related gap, and it fails more quietly.

plugins/openai/src/tts.ts requests pcm and then treats whatever comes back as raw samples:

  response_format: 'pcm',
  ...
  const buffer = await this.stream.then((r) => r.arrayBuffer());
  const audioByteStream = new AudioByteStream(OPENAI_TTS_SAMPLE_RATE, OPENAI_TTS_CHANNELS);
  const frames = audioByteStream.write(buffer);

There’s no decoder and no inspection of the response Content-Type. response_format is also hardcoded, so it can’t be changed through TTSOptions or updateOptions().

Against api.openai.com this is fine you ask for pcm, you get pcm. But baseURL is a public option in TTSOptions, and get provider() reads the host off this.#client.baseURL, so pointing this at an OpenAI-compatible server is clearly an intended use. If such a server doesn’t support pcm, or ignores response_format and answers with its own default, those bytes are written straight into the audio buffer as 16-bit samples.

Reproducing

  const client = new OpenAI({
    apiKey: 'test',
    baseURL: 'https://compatible.example.com/v1',
    maxRetries: 0,
    fetch: async () =>
      new Response(new Uint8Array(wav), { status: 200, headers: { 'content-type': 'audio/wav' } }),
  });
  const tts = new TTS({ client, model: 'kokoro' });

Serving a 9644-byte WAV (44-byte RIFF header + 9600 bytes of 24 kHz mono PCM) emits 9600 bytes whose first four bytes are the ASCII characters RIFF. The container header is played as audio; only a trailing partial frame is dropped.

For a WAV that happens to match the expected sample rate, the audible result is a short click before otherwise-correct audio, which is easy to miss. For a compressed format mp3, opus, aac, flac none of the body is PCM, so the whole utterance is noise. Either way there’s no error, no warning, and no retry.

Possible fix

Mirroring what just landed in Python: read Content-Type off the response and decode accordingly, falling back to the requested format when the server doesn’t declare something usable. Exposing response_format through TTSOptions would also help, since some compatible backends don’t offer pcm at all.

I’m happy to open a PR if you’d like I wrote the equivalent change for the Python plugin, so the shape would be familiar.

One note on test coverage

plugins/openai/src/tts.test.ts currently requires a live OPENAI_API_KEY and skips without one, so nothing exercises this path in CI. A small hermetic test using a stubbed fetch (as above) would cover it without network access.

Verified against @livekit/agents-plugin-openai 1.7.0.

2

Hi @abidullahcs.uk , one of our stated aims is to have the capabilities of the Agents JS project aligned with Agents. Since features / fixes usually land in the Agents repo first, we have a process which reviews these changes and updates Agents JS accordingly.

Please, that would be quicker :slight_smile: :rocket:

3

Thanks @darryncampbell
PR is up: fix(openai): reject TTS bodies that aren't raw PCM by abidullahdev · Pull Request #2330 · livekit/agents-js · GitHub

One deviation worth flagging: it rejects rather than decodes. agents-js has no streaming decoder agents/src/audio.ts only does file-based ffmpeg so mirroring the Python fix would mean building one first. This turns the silent noise into a clear APIError and exposes responseFormat; happy to look at a decoder separately if that’s the direction you want for alignment.

It’s waiting on workflow approval, since it’s my first PR to that repo.