Is there an STT time-to-first-partial metric (the ttft/ttfb analog)? Or am I missing it?

Agents
1

Trying to measure STT responsiveness per turn: the time from when the user starts speaking to the first interim transcript. Basically the STT analog of LLMMetrics.ttft and TTSMetrics.ttfb.

As far as I can tell there is no such field. STTMetrics has duration and audio_duration but no first-response metric, and duration is 0.0 for streaming STT. The only STT-adjacent latency I found is EOUMetrics.transcription_delay, but that measures the tail of the utterance (end of speech to the final transcript) and folds in the provider’s endpointing, so it can’t tell me how fast the first word actually appeared.

I know about the open requests for utterance_end_latency (#5063, #4795, PR #4966), but those are also the tail (end of speech to FINAL). What I’m after is the head:

span what it measures status
end-of-speech to FINAL finalization latency (transcription_delay / utterance_end_latency) exists / requested
speech onset to first INTERIM how fast the first word shows up ?

A couple of questions:

  1. Is there an existing way to get the speech-onset to first-interim latency today that I have missed?

  2. If not, would the team be open to exposing it? It looks computable at the session layer without touching provider timestamps: AudioRecognition already holds the VAD-backdated onset (self._speech_start_time) and consumes the INTERIM_TRANSCRIPT events in the same place, so it is just a subtraction the framework does not currently do. It also sidesteps the “providers report end_time unreliably” concern raised on #5063, since this never touches end_time.

For now I derive it myself by subtracting the user_state_changed -> speaking onset from the first user_input_transcribed(is_final=False), which works but is accurate only to the emit hop. First-class support would be much cleaner.

Happy to open an issue or draft a PR if there’s interest.

2

Here is what Ask AI in Documentation replied:

You’re not missing it. Today there is no built-in “speech onset to first interim transcript” metric analogous to LLMMetrics.ttft or TTSMetrics.ttfb.

The current metrics surface only exposes:

STTMetrics.duration, which is explicitly 0 for streaming STT
EOUMetrics.transcription_delay, which measures end-of-speech to final transcript
started_speaking_at / stopped_speaking_at timestamps on per-turn metrics
See Data hooks and Events and error handling [blocked]

Your workaround is effectively the correct approach today: correlate user_state_changed → speaking with the first user_input_transcribed(is_final=False) event. That’s the closest available proxy for STT first-token latency.

Your proposal also lines up well with the current architecture. Since the session layer already tracks VAD-backed speech onset and receives interim transcript events, a framework-level metric is computable without relying on provider timestamps. Opening an issue or PR for something like STTMetrics.first_interim_latency or speech_to_first_interim would be reasonable.

3

@Mahimai_Raja, You are not missing it. There is no STT first-interim metric today, and your read of the internals is correct.

  • STTMetrics only exposes duration (0.0 for streaming), audio_duration, and streamed. ttft is LLM-only, ttfb is TTS-only, no STT analog (metrics/base.py#L36).
  • transcription_delay is the tail: “after the end of the user’s speech” (#L101), computed as last_final_transcript_time minus last_speaking_time (audio_recognition.py#L114).
  • The head is a pure framework subtraction: the VAD-backdated onset (self._speech_start_time, #L1321) and the INTERIM_TRANSCRIPT branch (#L1271) are both in _on_stt_event, so no provider timestamps are involved.
# AudioRecognition._on_stt_event, INTERIM_TRANSCRIPT branch
elif ev.type == stt.SpeechEventType.INTERIM_TRANSCRIPT:
    if self._first_interim_at is None and self._speech_start_time is not None:
        speech_to_first_interim = time.time() - self._speech_start_time

Computing it there beats your user_input_transcribed proxy, which sits one layer downstream at the AgentSession emit and folds in that hop. A focused issue or a PR adding first_interim_latency to STTMetrics is the right step, and since it never reads provider end_time the #5063 reliability concern does not apply..

4

Thanks Muhammad, that confirms it and sharpens the implementation. Computing it in
_on_stt_event off self._speech_start_time at the first INTERIM_TRANSCRIPT is
cleaner than my user_input_transcribed proxy, which folds in the AgentSession
emit hop. I’ll use that.

One design point before a PR: the value is computed in AudioRecognition
(session layer), and transcription_delay (the tail) already lives on
EOUMetrics. Putting the head there too keeps both STT-timing halves on one
session-layer metric and matches where it’s computed. STTMetrics is the STT
plugin’s per-recognition metric and never sees the VAD onset, so surfacing it
there would need the framework to enrich the plugin metric. Any preference,
EOUMetrics.first_interim_delay vs STTMetrics.first_interim_latency?

Either way I’ll open a focused issue and happy to open a PR

5

Not exposed yet #6535 (Add first interim STT latency metric by spyrux · Pull Request #6535 · livekit/agents · GitHub) adds first_interim_delay to EOUMetrics, still open as of 1.6.6. It won’t land on STTMetrics; STT.emit never sees VAD onset.

You can get the exact number today without patching. started_speaking_at on the user message’s MetricsReport is the VAD-backdated onset, and the first is_final=False UserInputTranscribedEvent.created_at is stamped in the same call the PR latches on:

 first_interim_at = None

 @session.on("user_input_transcribed")
 def _on_transcribed(ev):
     global first_interim_at
     if not ev.is_final and first_interim_at is None:
         first_interim_at = ev.created_at

 @session.on("conversation_item_added")
 def _on_item(ev):
     global first_interim_at
     if ev.item.role == "user":
         onset = ev.item.metrics.get("started_speaking_at")
         if onset and first_interim_at:
             logger.info("first_interim_delay=%.3f", first_interim_at - onset)
         first_interim_at = None

Worth raising on the PR: it latches on _speech_start_time (per-burst) but subtracts against _user_turn_start (turn-level), so “Hello.” “…I need help” swallows the inter-burst silence.