Hey @AI_Tools thanks for the thorough write-up. Right now we do our routing based on the room id (proper prefix trie routing will follow). So right now, the speed benefits will be limited to live agent sessions. Were your tests running in a livekit room or done by directly calling the completions endpoint?
Neil_Dwyer
(Neil Dwyer)
6
Related topics
| Topic | Replies | Views | Activity | |
|---|---|---|---|---|
| Gemma 4 Que Times - LiveKit Inference | 9 | 216 | August 26, 2026 | |
| Livekit inference on self hosted | 1 | 77 | July 27, 2026 | |
| Gemini 3 Flash Preview via LiveKit Inference has much higher TTFT/jitter than direct Vertex in same Agents workflow | 1 | 61 | May 15, 2026 | |
| LiveKit Inference timeouts happening since past 2 weeks | 63 | 701 | August 27, 2026 | |
| Bad LiveKit Inference ttft for gpt-4.1 | 1 | 71 | April 15, 2026 |