Latency advice needed: OpenAI Realtime-quality conversation + Runway avatar via LiveKit Agent is too slow

Getting Started
1

Hey everyone,

I’m building a mobile app. The product lets a user talk to a pet as an emotional AI companion. The core experience needs to feel very responsive and emotionally present.

We have tested a few architectures:

  1. OpenAI Realtime only

    • Very good emotional conversation quality

    • Fast response time

    • Memory/persona works well

    • But the visual layer is not good enough yet

  2. Runway Characters direct mode

    • Very good animation quality

    • Conversation speed feels good

    • But Runway owns too much of the brain/voice/persona

    • Our Talking.Pet memory and personality control are not strong enough

  3. Current LiveKit Agent architecture

    • Flutter client publishes user mic into our LiveKit room

    • LiveKit Agent receives user audio

    • Agent uses OpenAI/LLM + TTS

    • Runway avatar plugin animates the agent audio

    • Flutter receives remote audio/video from the avatar

The current architecture is conceptually what we want:

User mic → Talking.Pet brain/memory/persona → Talking.Pet TTS → Runway avatar animation → Flutter

But the experience feels too slow and the emotional companion feeling disappears.

Some recent latency logs after optimization:

ENDPOINTING_MODE=dynamic
ENDPOINTING_MIN_DELAY=0.2
ENDPOINTING_MAX_DELAY=1.2
ENDPOINTING_ALPHA=0.8
TTS_PIPELINE=phrase_flush_not_sentence_buffer

Example turn: “How are you?”

SPEECH_END_TO_STT_FINAL_MS=1690
STT_LATENCY_MS=1690
LLM_FIRST_TOKEN_MS=569
TTS_LATENCY_MS=880
RUNWAY_PLAYBACK_BUFFER_MS=1
TOTAL_TIME_TO_FIRST_AUDIO_MS=3204
TOTAL_TURN_TIME_MS=5549

Example turn: “What is your name?”

SPEECH_END_TO_STT_FINAL_MS=1104
STT_LATENCY_MS=1104
LLM_FIRST_TOKEN_MS=811
TTS_LATENCY_MS=764
RUNWAY_PLAYBACK_BUFFER_MS=2
TOTAL_TIME_TO_FIRST_AUDIO_MS=2768
TOTAL_TURN_TIME_MS=5697

Example turn: “What’s the name of your mother?”

SPEECH_END_TO_STT_FINAL_MS=1150
STT_LATENCY_MS=1150
LLM_FIRST_TOKEN_MS=556
TTS_LATENCY_MS=1042
RUNWAY_PLAYBACK_BUFFER_MS=2
TOTAL_TIME_TO_FIRST_AUDIO_MS=2850
TOTAL_TURN_TIME_MS=5816

The good news is that the Runway playback buffer is now very low, often 1–2 ms. But the overall response still feels too slow because STT/endpointing + LLM first token + TTS first audio stack up.

We also saw occasional timing/turn-taking warnings like:

playback_finished called before text/audio input is done
push_audio called after close
skipping user input, speech scheduling is paused

Questions:

  1. Is this architecture expected to be slower because we are chaining multiple realtime systems together?

  2. Are there recommended LiveKit Agent settings for a more natural emotional companion / low-latency voice loop?

  3. Can endpointing be made more aggressive than dynamic min_delay=0.2, max_delay=1.2, alpha=0.8 without hurting reliability?

  4. Is there a better way to start TTS/avatar output earlier from partial LLM output?

  5. Are there known limitations when using the Runway avatar plugin for low-latency conversational use?

  6. Would a different LiveKit avatar plugin/provider be better suited for sub-2-second emotional conversation?

  7. Any suggestions for avoiding the playback synchronizer warnings above?

Our target is:

Time to first audible pet response: ideally 1–2 seconds
Very short emotional responses: 3–8 words
Animation must feel alive, but latency matters more than perfect lip-sync

We are currently considering moving live conversation back to OpenAI Realtime and using a local audio-driven pet cutout animation engine, while keeping Runway for offline/premium animation assets.

Before we make that architecture decision, I’d love to know if there are LiveKit-specific optimizations or better patterns we should try.

Thanks for any guidance.

2

@Niklas_Stalberg, one of the root causes: your STT_LATENCY (1100-1700ms) is the dominant cost in the turn budget. Endpointing min=0.2 / max=1.2 is already aggressive vs the defaults of 0.5 / 3.0 [ Turn-taking tuning | LiveKit Documentation ], and the model is using close to the full max_delay window, so the endpointing wait IS most of that 1.1-1.7s. Pushing max_delay lower trades latency for false-positive turn cuts; the documented move when cuts start happening is the turn-detector model rather than raw VAD timers.

The single biggest lever still on the cascade is preemptive_tts: true. Preemptive LLM generation is on by default (LLM starts when the final transcript arrives), but TTS waits for buffered phrases unless you opt in. preemptive_tts starts TTS on partial LLM output [same page]. That should shave a few hundred ms.

Runway is not your bottleneck. Your own metric shows the buffer at 1-2ms; switching avatars won’t move the needle.

For sub-2-second first audio you have to drop STT. The realtime path eliminates it. The half-cascade architecture (a realtime model handling ASR+LLM, with a separate TTS feeding the avatar) is the documented pattern for keeping your own TTS, persona, and memory in the loop while removing the STT cost [ Realtime models overview | LiveKit Documentation ].

The three timing warnings are separate from the latency problem and don’t need to be solved to hit your latency target. They’re agent-internal lifecycle issues (segment synchronization, session-close ordering, turn-scheduling state).

3

In addition to :up_arrow: , have you tried switching out your STT provider, your STT latency does seem quite high.

Is this architecture expected to be slower because we are chaining multiple realtime systems together?

Yes, this cascade model can introduce latency, so you need to be conscious of which parts of your pipeline are introducing delay. I usually point people here:

It honestly sounds like you are asking all the right questions, and identifying which part of your pipeline is introducing delays. There aren’t any known limitations with Runway avatars, but they are really new, we only launched them the other week.

4

Thanks for your help. Now the latency is low and acceptable once I am in the conversation with the agent. However startup is incosistent and very slow. Do you have any insight into the below on how I can improve it?

The issue is that total startup latency is often very high. I am trying to isolate whether this is caused by my implementation, Runway, LiveKit room creation, region selection, or agent dispatch behavior.

Some examples from recent logs:

Example 1 — first session:

BACKEND_SESSION_CREATE_MS=11454
LIVEKIT_ROOM_CREATE_MS=7925
LIVEKIT_DISPATCH_MS=1813
LIVEKIT_CONNECT_FROM_FLUTTER_MS=2050
AVATAR_JOIN_TO_VIDEO_PUBLISHED_MS=5465
TOTAL_APP_START_TO_FIRST_FRAME_MS=23188
BOTTLENECK=backend_session

Example 2 — pet switch / second session:

RUNWAY_BACKEND_REQUEST_MS=24646
BACKEND_SESSION_CREATE_MS=24583
LIVEKIT_ROOM_CREATE_MS=21973
LIVEKIT_DISPATCH_MS=1368

Example 3 — another pet switch:

RUNWAY_BACKEND_REQUEST_MS=23205
BACKEND_SESSION_CREATE_MS=23152
LIVEKIT_ROOM_CREATE_MS=20415
LIVEKIT_DISPATCH_MS=1408

The Python agent worker itself is already running and registered before the test:

registered worker
agent_name=pet-avatar
region=India South
livekit-agents version=1.5.15
rtc-version=1.1.8

The backend dispatches to:

agentName=pet-avatar

The worker does receive the job and joins the room, but the backend’s short join-poll often says:

LIVEKIT_AGENT_JOIN_CONFIRMED=false
RUNWAY_AGENT_JOIN_HANDSHAKE_WARNING=worker_may_join_after_flutter_connects

So I understand that the join-confirmation warning may be due to my polling logic being too short, and not necessarily a LiveKit failure.

However, the bigger concern is the room creation time:

LIVEKIT_ROOM_CREATE_MS=7925
LIVEKIT_ROOM_CREATE_MS=21973
LIVEKIT_ROOM_CREATE_MS=20415

Questions:

  1. Is it normal for LiveKit Cloud room creation to take 8–22 seconds?
  2. Could this be related to region selection or cold-start behavior?
  3. The worker is registered in India South. My app/backend are running locally from Thailand. Could region choice explain some of this delay?
  4. Is there a recommended way to pre-create or reuse rooms for low-latency agent experiences without keeping an expensive avatar session alive?
  5. Should I avoid explicit room creation and instead let the first participant connection create the room?
  6. Is there a better pattern for Agent Dispatch where the backend can return faster and the client connects while the worker joins?
  7. Is there a recommended health/readiness API for checking that a specific agent worker is registered before dispatching?
  8. Are there any LiveKit Cloud settings, region pinning options, or room creation patterns that would reduce startup time?

To be clear, I am not assuming LiveKit is the only cause. Runway avatar video publishing also adds latency. But the LIVEKIT_ROOM_CREATE_MS spikes look large enough that I want to confirm whether this is expected or whether my room/dispatch flow is suboptimal.

Any guidance on the recommended lowest-latency architecture for Flutter client → backend → LiveKit Agent → Runway avatar would be appreciated.

5

Can you share the session IDs for these examples. Room session id starts with RM_ and you can find them on your LiveKit dashboard under sessions.

Do you have Observability enabled? If so can you share the sessions with us?

6

@Niklas_Stalberg, The startup-latency split is illuminating. LIVEKIT_ROOM_CREATE_MS=7925-21973ms is your dominant startup cost, and that’s because you’re calling CreateRoom explicitly when you don’t need to.

Per the dispatch docs: “The room is automatically created during dispatch if it doesn’t already exist” [ Agent dispatch | LiveKit Documentation ]. Rooms also auto-create when the first participant connects [ Room management | LiveKit Documentation ]. The explicit CreateRoom call before dispatch is redundant. Drop it:

  # Before: 3 API roundtrips
  # 1. await lkapi.room.create_room(...)
  # 2. await lkapi.agent_dispatch.create_dispatch(...)
  # 3. Flutter connects

  # After: 2 roundtrips
  await lkapi.agent_dispatch.create_dispatch(
      api.CreateAgentDispatchRequest(
          agent_name="pet-avatar",
          room="my-room",  # auto-created by dispatch if missing
      )
  )
  # Flutter connects in parallel; whoever wins, the room exists.

That eliminates the 8-22s step. BACKEND_SESSION_CREATE_MS should drop in line.

On the Thailand→India South region mismatch: cross-region API roundtrips compound. Region pinning is for compliance traffic isolation, not latency optimization [ Region pinning | LiveKit Documentation ]. The cleanest fix is colocating your backend closer to the worker region.

The three timing warnings are separate from startup latency and don’t need to block your target.

7

Thanks again — your advice was very helpful.

I removed the explicit CreateRoom step and now let dispatch / first participant connection create the room. I also changed the backend so it returns the LiveKit token before dispatch completes, and I deferred the worker registration probe out of the critical path.

That fixed the original major backend bottleneck.

Before:

LIVEKIT_ROOM_CREATE_MS=7925–21973
BACKEND_SESSION_CREATE_MS=11454–24583
TOTAL_APP_START_TO_FIRST_FRAME_MS≈23188

Now the backend fast path is much better:

LIVEKIT_ROOM_CREATE_SKIPPED=true
LIVEKIT_DISPATCH_AUTOCREATE_ROOM=true
LIVEKIT_DISPATCH_ASYNC=true
LIVEKIT_DISPATCH_AWAITED=false
BACKEND_RETURNED_BEFORE_DISPATCH_DONE=true
BACKEND_FAST_RETURN_MS=39–59
BACKEND_TOTAL_SESSION_MS=39–59

Flutter now also starts the backend request early:

FLUTTER_PROFILE_READY_TO_BACKEND_REQUEST_MS≈298–394
FLUTTER_BACKEND_PRE_SEND_MS≈4–5
FLUTTER_BACKEND_REQUEST_MS≈1614–1775

So the original room-create/backend problem is much improved.

The remaining issue is cold-start variance before the first Runway avatar video frame appears. Recent runs are roughly:

Best recent run:
FIRST_DOG_TOTAL_TO_FIRST_FRAME_MS=13566
FIRST_DOG_BACKEND_MS=1614
FIRST_DOG_LIVEKIT_CONNECT_MS=3377
FIRST_DOG_WORKER_TOTAL_TO_VIDEO_MS=7455
FIRST_DOG_VIDEO_PUBLISHED_TO_FIRST_FRAME_MS=759

Slower recent run:
FIRST_DOG_TOTAL_TO_FIRST_FRAME_MS=16993
FIRST_DOG_BACKEND_MS=1775
FIRST_DOG_LIVEKIT_CONNECT_MS=2452
FIRST_DOG_WORKER_TOTAL_TO_VIDEO_MS=11325
FIRST_DOG_VIDEO_PUBLISHED_TO_FIRST_FRAME_MS=981

The worker is already running and registered before the test. It receives the dispatch and joins the room. The worker now starts OpenAI Realtime setup and Runway avatar setup concurrently:

WORKER_OPENAI_AND_RUNWAY_SETUP_PARALLEL=true
WORKER_SETUP_START_DELTA_MS=0
WORKER_ROOM_JOIN_API_MS≈2400
WORKER_PARALLEL_SETUP_TOTAL_MS≈2800–4000
WORKER_TOTAL_TO_VIDEO_PUBLISHED_MS≈5200–6500
RUNWAY_CHARACTER_FALLBACK_USED=false
RUNWAY_AVATAR_JOIN_RETRY_ENABLED=false

Once the session is live, the conversation latency is good:

TOTAL_TIME_TO_FIRST_AUDIO_MS≈775–1044
RUNWAY_AGENT_STREAM_BUFFER_MS≈2–5

So the main remaining question is startup, not turn latency.

Questions:

  1. With this architecture — Flutter client → backend → LiveKit Agent Dispatch → Python LiveKit Agent → Runway avatar participant — is ~13–17s cold start to first avatar video frame expected?

  2. Is there a recommended pattern to reduce the remaining cold-start variance without keeping the expensive Runway avatar session alive all the time?

  3. Should the backend dispatch first and return token immediately, as I am doing now, or is it better for the Flutter client to connect first and then trigger dispatch?

  4. Is there a recommended way to prewarm the LiveKit Agent / room / dispatch path without starting the Runway avatar session yet?

  5. In this kind of avatar flow, should I expect the largest remaining cost to be Runway avatar participant join + video track publish, rather than LiveKit itself?

  6. Are there any LiveKit Cloud region / worker deployment recommendations for this pattern? My worker is currently registered in India West and I am testing locally from Thailand.

  7. Is there any reliable agent-readiness API I should use before dispatch? I tried probing worker registration, but the probe failed and was not useful, so I removed it from the critical path.

Current conclusion on my side:

  • Removing explicit room creation was absolutely the right fix.

  • Backend startup is now fast.

  • The app is stable.

  • The remaining bottleneck appears to be worker/avatar/video publication coordination.

  • I am trying to understand whether there is still an architectural improvement available, or whether the right product solution is to introduce a Dog Home screen and prewarm/prepare the session before the user enters the conversation.

Any advice on the best-practice architecture for lowest perceived startup latency with LiveKit Agents + Runway avatars would be appreciated.

8

@Niklas_Stalberg, Glad the room-create fix landed. Your metrics now point cleanly at the real remaining cost: WORKER_TOTAL_TO_VIDEO at 7.5-11.3s is the avatar spinning up and publishing its first frame, not LiveKit room/dispatch/connect (now your small legs).

That spin-up is per session by design. The avatar launches when the agent calls avatar.start(session, room=ctx.room) and the provider joins as a separate participant to publish the audio+video [ docs.livekit.io/agents/models/avatar ]. prewarm_fnc won’t touch it, it inits the worker process before the job, not the per-room avatar session [ livekit/agents worker.py ]. So there’s no way to pre-spin the avatar from the worker side.

Which makes your Dog Home screen instinct the right call: you can’t delete the 7-11s, so overlap it. Dispatch the agent (which fires avatar.start) the moment the user hits the pet-select screen, so the avatar is already live before they enter the conversation. That’s how you hide avatar cold start.

Region is a minor lever here: India West worker vs Thailand client mostly hits your connect leg (2.4-3.4s), not the avatar launch. And you can drop the readiness probe, dispatch auto-creates and the worker joins async, so gate the UX on the prewarm screen, not a poll.