Prompt caching for google/gemma-4-31b-it on LiveKit Inference how to get cached input?

Getting Started
1

Prompt caching for google/gemma-4-31b-it on LiveKit Inference how to get cached input?

2

There’s no flag to switch on. Read prompt_cached_tokens off LLMMetrics in your metrics_collected handler, or input_cached_tokens if you’re aggregating with UsageCollector. That’s where the cached input count comes through.

Worth sanity checking it’s actually non-zero though. LiveKit’s pricing does list “Input (cached)” separately for Gemma 4 at $0.20/1M, so it’s real and billed, but the LLM docs don’t document any caching controls.

3

Right that there’s no flag, but the UsageCollector half needs correcting, and the reason there’s no flag is the actually useful part.

The fields

Per inference call: LLMMetrics.prompt_cached_tokens, described in the docs as the number of cached tokens in the input prompt. Read it from a per-plugin metrics_collected handler. Note the session-level metrics_collected event is deprecated; the per-plugin one is not.

Cumulative per session: session.usage, where each LLMModelUsage entry carries input_cached_tokens, plus input_cached_text_tokens, input_cached_audio_tokens and input_cached_image_tokens for the multimodal breakdown.

Live during the session: subscribe to session_usage_updated.

UsageCollector will not give you this. It is deprecated in favour of session.usage, and its UsageSummary only ever held llm_prompt_tokens, llm_completion_tokens, tts_characters_count and stt_audio_duration. There is no cached field on it at all, so that path returns nothing regardless of deprecation.

@session.on("session_usage_updated")
def _on_usage(ev):
    for u in ev.usage.model_usage:
        if hasattr(u, "input_cached_tokens"):
            print(f"{u.provider}/{u.model}: "
                  f"{u.input_cached_tokens}/{u.input_tokens} cached")

Why there is no flag to turn on

Gemma 4 31B is served on LiveKit’s own GPUs behind SGLang, per their engineering post on the launch. SGLang does automatic prefix caching at the serving layer, so this is implicit KV prefix caching, not an explicit cache-control API like Anthropic’s cache_control blocks or Gemini’s explicit cached content.

That distinction is the whole answer to your question. There is nothing to enable. Cache hits are something your prompt earns by presenting a byte-identical prefix to a request the node has already seen.

How to actually get hits

Prefix caching matches from the start of the prompt and stops at the first difference. So:

Order matters more than anything else. Static content first: system instructions, tool definitions, few-shot examples, any retrieved context that does not change within the session. Volatile content last.

Things that silently zero out your hit rate, all common in voice agents:

  • A timestamp, date, or session ID in the system prompt
  • The caller’s name or phone number injected at the top rather than appended
  • Tool definitions serialized in nondeterministic order
  • Per-turn RAG results prepended instead of appended
  • Any chat_ctx mutation that rewrites earlier turns, including truncation, rolling summarization, and agent handoffs that rebuild context

The last group is the one that catches people. A voice conversation grows monotonically, so turn N+1’s prefix is exactly turn N’s context. That is the ideal shape for prefix caching and it is why voice agents should see hit rates climb steadily through a call. But anything that rewrites history mid-session invalidates the prefix and resets you to zero, and you will not notice unless you are logging it.

Turn one always misses. That is expected, not a bug.

Sanity check it

Log both numbers per turn and watch the ratio:

llm.on("metrics_collected", lambda m: print(
    f"{m.prompt_cached_tokens}/{m.prompt_tokens}"
))

Across a normal conversation the cached share should rise turn over turn. If it is still zero by turn five, something in your prefix is changing every request and the list above is where to look.

Worth asking LiveKit directly

Three things that matter for cost modelling and none of which appear to be documented:

  1. Is caching best-effort or guaranteed for this model?
  2. What is the eviction window or TTL?
  3. Is the cache node-local? With a GPU fleet, the prefix cache lives on whichever node serves the request. If requests are not pinned to a node for the duration of a session, hit rates would suffer in a way no amount of prompt engineering fixes.

Worth checking the current cached rate on the pricing page rather than quoting a figure from memory, since Inference rates move.