Making Voice Agents sound human with Expressive Mode

Announcements
1

Agents have gotten very good at knowing what to say. How they say it hasn’t kept up.

Tell a voice agent your flight was cancelled and you’re stranded overnight with two kids. Most will confirm your rebooking in the same bright tone they use to upsell seat selection.

Expressive mode is now live in LiveKit Agents. Just flip one flag and your LLM starts emitting prosody markup inline (emotion, pacing, pauses, laughs), which we strip before your transcript renders.

Streaming sentence-by-sentence to TTS kills expressiveness. Each arrives without context, pitch drifts between them. Batching into larger chunks holds the emotional through-line, with no noticeable latency hit.

Users don’t want maximum emotion, they want appropriate emotion. The customer service preset deliberately doesn’t mirror an angry caller, it leans calm instead.

Voice AI that gets the words right but the feeling wrong still loses users. Expressive mode is our first step toward fixing that. Hear the difference (check out our demo in the comments).

Watch the demo:

Announcement blog:

Resources

6 Likes
2

This doesn’t work well when we have multiple agents, since some agents need to be expressive while others should remain neutral.

Currently, we handle this by dynamically injecting a TTS voice addendum based on the current provider.

There’s also another issue: our TTS is implemented as shown below. We’ve tried enabling expressiveness at the agent-session level as well, but that didn’t work.

def _make_tts():
        return inference.TTS(
            model=f"fishaudio/{(c and c.model) or 's2.1-pro'}",
            voice=(c and c.voice) or "59b8d12e87724010a34190047391be4f",
            extra_kwargs=extra_kwargs,
        )

session = AgentSession(
        userdata=userdata,
        expressive=True # i have added but still dont work
    )

class TestAgent(Agent):
     def __init__(self, userdata: ) -> None:
        super().__init__(
            stt=stt,
            tts=tts,
            llm=llm,
            vad=vad,
            instructions=_build_instructions(),
            expressive # not allowed here, its only available in agent session level
        )
3

Hi Kaushal,

This doesn’t work well when we have multiple agents, since some agents need to be expressive while others should remain neutral.

I’ll raise this with the engineering team, I believe there were technical reasons that expressive mode was applied at the agent session level, but I’m unsure what the future roadmap is. I understand the use case.

We’ve tried enabling expressiveness at the agent-session level as well, but that didn’t work.

How were you testing? I just tested with the example agent source and specifying { "expressive": true, "tts": "fishaudio" } in the agent job metadata had a clear difference compared to specifying { "expressive": false, "tts": "fishaudio" } . You can also see <expr agent debug output in the former.

1 Like
4

Thanks, i will try again this week and maybe share sample snippet and will check above example agent again to see if i miss anything in config.

5

Hi @Kaushal_Shah, thank you for trying out expressive mode and for your feedback! I’ve opened a PR here to support dynamic expressive updates. Please feel free to try it out and let us know what you think :raising_hands:

2 Likes
6

Hi @Tina_Nguyen Thanks so much! I’ll give this a try as early as possible. Much appreciated