Audio Simulations: Which TTS Model Powers the Simulated User, and How Is the Simulated User Billed?

Agents
1

Hi LiveKit team,

I’m evaluating the new audio simulation feature for Agents and had a couple of questions about the simulated user voice pipeline.

In audio mode, the docs mention that the simulated user “speaks your scenario aloud” and joins the room as a participant, but I could not find details about the implementation of that synthesized voice. It user STT generated on the agent, at livekit servers or at the site of the simulation CLI command? I assume livekit servers from what I can see.

Specifically:

  1. What TTS model/provider is used for the simulated user’s speech in audio simulations?
  2. Is the simulated user’s synthesized audio billed separately from the agent’s own STT/TTS usage?
  3. If yes, how is that billing represented? Is it:
    • included in “audio turns”
    • billed as provider TTS usage
    • bundled into simulation pricing
    • or handled another way?
  4. Is there any way to configure or override the simulated user voice/model?

Relevant docs page: Agent simulations | LiveKit Documentation

Thanks!

2

What TTS model/provider is used for the simulated user’s speech in audio simulations?

I am not sure exactly how the simulation works and I don’t believe the team has disclosed that so I probably should not either.

Is the simulated user’s synthesized audio billed separately from the agent’s own STT/TTS usage?

We bill the entire simulation per-turn rather than just paying the sum(inference costs). (currently free)

Is there any way to configure or override the simulated user voice/model?

Can you tell me more about the usecase for why you need this or why it would be useful?

3

We bill the entire simulation per-turn rather than just paying the sum(inference costs). (currently free)

I see, interesting. I guess free whilst in beta

Can you tell me more about the usecase for why you need this or why it would be useful?

Regarding the voice model - it is partly just curiosity, but also because we have had challenges in past trying to get voice models to work well in Japanese, I haven’t done enough testing with the LiveKit simulated TTS to know if those problems apply, but if I knew what model it was that might save me some testing there.

Regarding the simulated LLM and LLM-as-a-judge, aside from curiosity, it would be useful to know as it could impact how complex a prompt we can issue to describe the simulated user behaviour and judge criteria (and also how much support for Japanese we can expect). Again I should note I haven’t put the Livekit Simulations through in depth scenarios yet, so I do not yet know if it would meet our requirements, but I noticed that it seems some of the metrics in the evaluation might not be supported in Japanese like Word Error Rate etc.

(I should also note after writing this post I sent a small support email asking a few more technical questions specific to a simulation (SR_YmqoN4WyEb52), to better understand the output I was seeing when I did some test runs, and to confirm how much of the “not measured” might be due to the language choice.)