Livekit supports ElevenLabs TTS models, but why not Eleven v3

Agents
, , ,
1

If I look at ElevenLabs TTS | LiveKit Documentation I can see the list of elevenlabs TTS models, I do not see Eleven v3, as it is a really rich TTS model with multiple languages support: Models | ElevenLabs Documentation

Is there any reason to not support it through Livekit inference?
First I though it can only be used with dialouge API, but in the doc it states:

With Eleven v3 comes a new Text to Dialogue API, which allows you to generate natural, lifelike dialogue with high emotional range and contextual understanding across multiple languages. Eleven v3 can also be used with the Text to Speech API to generate natural, lifelike speech with high emotional range and contextual understanding across multiple languages.

When I tried with:

tts=inference.TTS(
            model="elevenlabs/eleven_v3", voice="Xb7hH8MSUJpSbSDYk0k2", language="bn"
),

I get this error:

provider instance for session 4b1dc706-00d3-49ce-a197-749c9be56be0: TTS model not found in list"}, retrying in 2.0s

It would be really helpful if the Livekit team can help me here. Thanks in Advance!

2

Hi, I see internal discussions from a couple of days ago saying that the model will be added to Inference and the work is still ongoing.

I’m not sure what the latency will be, I see a lot of reports online that the model does not support WebSockets, so I assume model integration will rely on HTTP streaming.

There is a community-contributed PR to add v3 support to the plugin, but at this point you are probably best to wait until it is added to Inference.

3

Isn’t latency for v3 ~10s for a few lines of dialogue? That’s from my experience

1 Like
4

hey, I already created a pr for it: feat(elevenlabs): add eleven_v3 model support with HTTP streaming fallback by yusuf-eren · Pull Request #4936 · livekit/agents · GitHub

related issue: Proper support for ElevenLabs "eleven_v3" in "livekit-plugins-elevenlabs" with ETA · Issue #4901 · livekit/agents · GitHub

Also you can just bypass it without waiting for LiveKit Agent new version: Proper support for ElevenLabs "eleven_v3" in "livekit-plugins-elevenlabs" with ETA · Issue #4901 · livekit/agents · GitHub

1 Like