UsageSummary: per-model LLM/STT/TTS usage for cost calculation

Agents
1

What’s the update on this:
UsageSummary: per-model LLM/STT/TTS usage for cost calculation · Issue #4564 · livekit/agents

Was supposed to be shipped with v1.4 but it wasn’t :frowning:

2

So, this work was completed under automatically collect session usage by davidzhao · Pull Request #4566 · livekit/agents · GitHub and as you say, DZ said this would be shipped in 1.4.

As best I can parse the PR conversation, the change was bound up with another feature which, although originally planned for 1.4, was moved to 1.5. I do see this change in the RC for 1.5, livekit-agents · PyPI .

Apologies for the delay, hopefully this will be shipped in a general release very soon, not just an RC.

3

Hey @darryncampbell, thanks for looking into this, but I think there might be some confusion here. I’ve been digging into the actual state of main and the per-model usage tracking from PR #4566 doesn’t appear to have landed.

Specifically, what that PR introduced was:

  • A new metrics/usage.py with ModelUsageCollector, LLMModelUsage, TTSModelUsage, STTModelUsage – the classes that actually aggregate usage per provider/model pair (which is what you need for cost calculation)

  • Token billing fields (input_tokens/output_tokens) on STTMetrics and TTSMetrics for providers like OpenAI that bill per token

  • session_duration on RealtimeModelMetrics for session-based billing (xAI)

  • Deprecation of the old flat UsageSummary/UsageCollector

None of that is on main right now. The metrics/usage.py file doesn’t exist at all, STTMetrics/TTSMetrics don’t have token fields, and UsageSummary/UsageCollector have no deprecation warnings. What’s currently shipping is still the old flat collector that lumps all models into a single bucket – which isn’t useful for cost calculation since you can’t distinguish e.g. GPT-4o tokens from Claude tokens, or Deepgram audio from Whisper audio.

It’s possible the merge got partially reverted or the feature was decoupled from the 1.5 RC. Could someone from the team clarify whether this is planned for 1.5 GA or if it slipped to a later release?

4

It looks like the changes are in this commit: automatically collect session usage (#4566) · livekit/agents@df860f1 · GitHub

Which was merged into chenghaomou/v1.5.0, GitHub - livekit/agents at chenghaomou/v1.5.0 · GitHub

Since the officially released plugin is only at 1.4.5 I wouldn’t be surprised if those changes aren’t yet in main.

It’s possible the merge got partially reverted or the feature was decoupled from the 1.5 RC. Could someone from the team clarify whether this is planned for 1.5 GA or if it slipped to a later release?

You should be able to test this with livekit-agents · PyPI (there has been another RC released since my link in the previous answer)