Hi,
We’re building voice agents on livekit-agents (1.6.5) with google/gemma-4-31b-it via LiveKit Inference. The launch post advertises 192ms TTFT, but on the shared gateway (agent-gateway.livekit.cloud) we’re seeing about a 400ms floor, rising toward a second at realistic voice context sizes. To figure out how much is the model and how much is the serving path, we served the identical checkpoint ourselves on one H100 with stock vLLM. Numbers, recipe, and questions are below. Happy to share all the probe scripts.
How we measured: TTFT is the time from sending the request to the first SSE content chunk, with stream=true. Single warm connection, no concurrency, no retries. Network isn’t the bottleneck, since TCP connect to the edge is about 12ms. For cache probes we prepend a unique nonce (guaranteed cold) then send the byte-identical prompt again, which should hit.
Shared gateway vs self-hosted (median TTFT):
Tiny ~30-token prompt (warm floor): ~390ms vs 28ms
Fresh 1K prompt: ~500ms vs 130ms
Fresh 5.5K cold prefill: ~865ms vs ~864ms (identical, so pure compute, which is a nice sanity check)
Repeated 9K prefix: ~720ms vs 67ms (the cache actually hits when self-hosted)
50 concurrent voice sessions on the one H100: p50 73ms, p95 113ms, zero errors
Two things stand out. There’s a floor of roughly 350 to 400ms even on a tiny warm request, and with only 12ms of network in the path that looks like internal routing or queueing rather than compute. The launch post itself says warm requests start returning tokens in about 100ms. Second, prefix caching barely helps. Repeating a 9K prompt saves only about 100ms, which suggests requests land on different replicas with no session affinity. For a voice agent, the system prompt plus growing history is a stable prefix that just gets re-prefilled nearly every turn.
To be clear, this is only about latency. Quality was excellent on both endpoints. Our full 734-case eval covering tool calling, multilingual turns, and digit collection passed at 99.86 percent on each.
Our self-hosted recipe (vLLM 0.25.1, H100 NVL, agent co-located):
VLLM_USE_FLASHINFER_SAMPLER=0 \
vllm serve google/gemma-4-31B-it \
--dtype bfloat16 --quantization fp8 \
--enable-prefix-caching --enable-chunked-prefill \
--max-model-len 32768 --gpu-memory-utilization 0.90 --max-num-seqs 64 \
--tool-call-parser gemma4 --enable-auto-tool-choice \
--limit-mm-per-prompt '{"image":0}' --api-server-count 4
A few things we learned the hard way:
FP8 applied at serve time only, from the BF16 checkpoint. We never load pre-quantized weights. It was quality-neutral for us and mainly buys prefill speed.
Don’t pass --reasoning-parser gemma4. With it on, confirmation and correction turns occasionally ran past the end of the turn and fired a premature confirm tool call, so the task never completed. Dropping it fixed this completely, and tool calling still works with just the tool parser plus auto choice. Same on 0.24.0 and 0.25.1.
Set max_tokens to at least 256 on tool-call turns. At 100, a turn with two corrections at once emits two parallel calls, the second truncates mid-JSON, and the args come back full of schema placeholders. Threshold is around 192. We saw this on both endpoints.
The limit-mm-per-prompt and api-server-count 4 flags are optional. They only helped about 25 percent on synchronized bursts.
FLASHINFER_SAMPLER=0 because our box has no nvcc, and the default sampler tries to JIT a kernel at startup, which kills engine init.
Keep the system prompt byte-stable and send one warmup request at deploy time. That’s where the 67ms comes from.
Questions:
- Is 192ms achievable on the shared tier, or is it specific to dedicated deployments? If dedicated, what’s the path to that (docs or pricing)?
- Are there plans for session affinity or cross-turn prefix reuse on the shared gateway? For voice this is the single biggest lever, since re-prefilling costs 500 to 900ms per turn at realistic sizes.
- Is there any way today to pin a region or replica per session, maybe via headers, that we missed?
- Is the 350 to 400ms floor on tiny warm requests expected for the shared tier?
Charts attached: TTFT comparison, the prefix-caching comparison, and per-turn TTFT for the 50 concurrent sessions. Probe scripts are single-file Python, happy to post.
Thanks!
