I am using livekit.agents.inference.TTS with model="xai/tts-1", not livekit.plugins.xai.TTS. I would like to know which xAI TTS parameters the inference gateway actually forwards.
xAI documents these as WebSocket query parameters on wss://api.x.ai/v1/tts:
optimize_streaming_latency: 0, 1, or 2. 0 is the default and keeps the largest first chunk. 2 is the smallest first chunk.
speed: speaking-rate multiplier, about 0.7 to 1.5.
text_normalization: bool.
The direct plugin sends those as query parameters. That was livekit/agents#6095, which closed livekit/agents#6077. The Inference docs for xai/tts-1 only list model, voice, and language. They do not mention those three.
The Inference client still accepts them. livekit/agents#6469 added optimize_streaming_latency: Literal[0, 1, 2] to XaiOptions. speed, bit_rate, and text_normalization are on that TypedDict too. The SDK does not turn them into query parameters. It copies extra_kwargs into the JSON extra field on session.create and on each input_transcript packet to the inference gateway.
Does the inference gateway read extra.optimize_streaming_latency, extra.speed, extra.text_normalization, and extra.bit_rate and pass them to xAI?
If yes, which packet matters, session.create or input_transcript? Is 2 accepted, or only 0 and 1?
If no, please treat this as a feature request. Inference should accept the same xAI streaming options the direct plugin already forwards, and the Inference docs should list them. optimize_streaming_latency is the one I was lookign to try to avoid stuttering in TTS. With the default, generated audio falls behind realtime and LiveKit logs flush audio emitter due to slow audio generation, which is an audible cut in playback.
I suspect its due to STT playback being slower than it can be streamed, and does not occur when I use the same string on https://console.x.ai/team/5d42776b-1d07-4093-9b76-2f8523a0b8c2/voice/text-to-speech (which is not streamed presumably). So that is why I am keen to enable optimize_streaming_latency but I suspect it currently is not enable-able.
@Benjamin_Lowe, Your client-side trace is exactly right. XaiOptions.optimize_streaming_latency is Literal[0, 1, 2], and the inference SDK copies extra_kwargs into the extra field on both the session.create packet (inference/tts.py#L538-L541) and every input_transcript packet (#L688-L691). So the fields do leave the SDK on both.
What happens after that is not answerable from open source. The inference gateway is Cloud-side, not in livekit/agents, so whether it reads extra.optimize_streaming_latency, speed, text_normalization, or bit_rate and forwards them to xAI, which packet wins, and whether 2 is accepted are all gateway-internal. No issue tracks this yet, so filing the feature request you drafted is the move, and it gets the authoritative answer from the people who own the gateway (new issue).
To actually exercise the flag now, run this one agent through the direct plugin instead of Inference. livekit.plugins.xai.TTS takes these and puts them on the xAI WebSocket URL as real query parameters (xai/tts.py#L142-L149):
from livekit.plugins import xai
# reads XAI_API_KEY from env (its own key, not the gateway's); same voice/language you already use
tts = xai.TTS(voice="carina", language="ja",
optimize_streaming_latency=2, text_normalization=False)
Set one expectation before pinning hopes on it. By xAI’s own definition (0 largest first chunk, 2 smallest), optimize_streaming_latency tunes first-chunk size, not sustained generation rate. flush audio emitter due to slow audio generation is a mid-utterance underrun, generation falling behind realtime playout, so if xAI’s sustained rate for that Japanese text is below realtime, a smaller first chunk starts sooner but may not close the gap. The direct-plugin run will tell you quickly whether it helps.
Oh, maybe @Muhammad_Usman_Bashir answered your question? I assumed you meant off the backend of the inference gateway not from agent to gateway. I will still validate if those get forwarded to the peer service.
It would help to narrow it down with a test like this:
Set optimize_streaming_latency to 0, or leave it out, and see whether the early-utterance gaps go away.
Run the same test on a wired connection, or from a small cloud VM near Frankfurt (your room is hosted there). If the gaps disappear, the delay is on the way to your machine.
Try the same Japanese text with livekit.plugins.xai.TTS directly. If it gaps there too, it’s xAI’s streaming throughput for Japanese and we’ll raise it with them.
@CWilson and @Muhammad_Usman_Bashir thank you for your response. I tried setting optimize_streaming_latency to absent, 0, 1 and 2 and the issue persists with Livekit Inference XAI TTS.
I should note that “flush audio emitter due to slow audio generation” I think may have beed a “red herring”, as most of the time this issue occurs without that issue. I also tried setting os.environ.setdefault("LIVEKIT_AGENTS_LOOP_BLOCK_WARN_MS", "20") so see if its an asycio block related, but no warnings are logged.
You suggested trying this on a cloud VM near Frankfurt - which seems challenging as it would need to be able to route microphone traffic etc. but I can rule our my internet connection being too slow (wired connection) given if that was the problem then XAI plugin directly would surely have the same issue.
In this case, I guess the conclusion may be just that livekit inference is unable to offer the same streaming bandwidth to my location as XAI?
I attach another example sessions from my local machine in UK using the livekit inference with optimize_streaming_latency omitted. In this example, almost everytime the agent says the first word in an utterance (e.g. 承知いたしました in the screenshot) it gets a small gap between the first and second character.
If there is no workaround, it may prevent us adopting the Expressive mode from Livekit (at least with this TTS provider) which is a shame, but it does seem to be specific to Livekit Inference itself. The odd thing to me is that other than these tiny glitches, everything else seems healthy (no high latency to respond etc.) : if it was related to Livekit Inference having slow bandwidth to my particular machine’s ip, I would have thought it would cause havoc in other parts of the pipeline.
i’ll check with my colleagues in Japan if this issue is specific to me in the UK (and Frankfurt based Livekit Room). How does the regionality of livekit inference work - is it always the same region as the room? I wonder which region is used by Livekit Inference when the room is hosted in Tokyo.
I notice these gaps in speech like I screenshotted and the other I shared above are often just under 20ms actually, so this doesn’t prove much, and 20ms is the lowest this env var goes
I wonder if when running livekit inference it’s possible something is blocking event loop for 19ms every now and then? But if that was the case I can’t imagine I would be the only one reporting it!
@Benjamin_Lowe I dug into this and created tests that reproduce it. I looked through the agent code and believe I’ve found the root cause. I am going to pass this off to our agents team to validate my results.
At a high level, I ran the test directly against the inference service (no agent framework) and didn’t see the issue. I did the same test with the agents framework and was able to reproduce the issue you exposed here.
