Realtime metric calculation issue

Agents
,
1

Approach 1 :

user_state_changed: We used the speaking → listening state transition as T0 (created_at) and the assistant’s started_speaking_at as T1, calculating E2E = T1 − T0 for user_reply turns.

Why it failed: With Gemini Live without session VAD, listening was tied to the plugin’s input_speech_stopped event at turn completion, which could fire after the bot had already replied rather than when the user actually stopped speaking. This made T0 late or stale across turns, sometimes pairing the wrong start timestamp with the next T1. The calculation was correct, but the start-time anchor was wrong.

Observed values: E2E was inflated, often around 6–15 seconds per turn in latency_ms_turns (occasionally spanning roughly 1–15 seconds), rather than the expected ~0.5–1.5 seconds. Meanwhile, RealtimeModelMetrics TTFT remained very low—often around 1–6 ms median—because it reflects the Google plugin’s generation/transcript-related metric, not actual user-perceived response latency.

Approach 2

user_input_transcribed: We set T0 to the created_at of the latest non-final user_input_transcribed event at or before T1, kept the assistant’s started_speaking_at as T1, and calculated E2E = T1 − T0 for user_reply turns.

Why it failed: On these Gemini Live calls, input text typically arrived as a single chunk just before the first agent audio, rather than as partial transcripts while the user was still speaking. Consequently, T0 landed close to T1 instead of marking the acoustic end of the user’s speech. The subtraction was correct, but the T0 proxy was wrong: we measured transcript arrival to playout, not user speech-end to playout.

Observed values: E2E was deflated, often around 10–300 ms per turn; one session included values such as 28, 11, 10, 44, 187, and 137 ms, while another had a median of approximately 32 ms, with many turns between 5 and 350 ms. RealtimeModelMetrics TTFT was still misleadingly low, with medians around 2.8–6.4 ms, because the Google plugin’s metric is tied to its transcript/generation pipeline and does not represent true model response latency or user-perceived delay.

i have tried two apporaches for the Gemini 3.8 live model. i have got the metrics wrong which has show the ttft around 10ms to 300ms and similar end to end latency. i think might also happen with openai aslo.

2

Hi, the best way to calculate realtime metrics would be to use the session reports:

which provide you with all the required metrics, such as TTFT for each turn.

If you are comparing TTFT with published figures, make sure you are comparing like-for-like, e.g. often published figures will state the P50 figure. The session reports, passed through AI, can also help you understand any anomalies, for example perhaps your sessions have a high number of tool calls which are affecting your metrics.

3

Your first anchor is late because, without a session VAD, the user’s “listening” state just follows Gemini’s input_speech_stopped event, timed when the event arrives.

Pass the session its own VAD:

session = AgentSession(llm=google.realtime.RealtimeModel(...), vad=silero.VAD.load())

Gemini still decides when to reply. But the end-of-speech time now comes from Silero, back-dated by the silence it waited for. Heads up: Silero can now also interrupt the agent when the caller talks over it, so check barge-in still feels right.

Then you don’t have to compute anything. The assistant messages carry it:

@session.on("conversation_item_added")
def _(ev):
    if ev.item.role == "assistant" and "e2e_latency" in ev.item.metrics:
        print(ev.item.metrics["e2e_latency"])

That’s first audio frame out minus when the user stopped speaking. It’s missing on turns the agent starts without new user speech, like the greeting or a generate_reply. The plugin’s TTFT is its own internal timing, not what the caller hears.

This is from reading the livekit-agents 1.8.9 source (pip show livekit-agents to check yours). I haven’t run it against Gemini Live, so compare a few turns with a recording first.