LiveKit Inference STT returns HTTP 429 on single isolated sessions despite available concurrency

Agents
1

Hello LiveKit Support,

I am experiencing a reproducible issue with LiveKit Inference STT on project:

Project ID: p_368s6zk11dj
Agent ID: CA_bN3Jku5rMugd

The issue is reproducible both in outbound telephony and directly in LiveKit Test in Console, so SIP, Twilio, Vercel and my application orchestration can be excluded.

Observed behavior:

  1. AssemblyAI Universal-3.5 Pro
    Model ID:
    assemblyai/universal-3-5-pro

Outbound job:
AJ_bL7ArtycpyhG

On 22 Aug 2026, between approximately 15:48:46.556 and 15:49:04.542 UTC, the STT stream repeatedly failed with:

APIStatusError: Invalid response status (429 Too Many Requests)

The traceback showed the LiveKit internal inference gateway URL:

ws://inference.livekit.svc.cluster.local/v1/stt?model=assemblyai/universal-3-5-pro

After repeated retries, the AgentSession closed due to the unrecoverable 429 error before the SIP call was answered.

  1. AssemblyAI Universal Streaming Multilingual

I then changed the STT model to:

assemblyai/universal-streaming-multilingual

and tested directly in LiveKit Test in Console, without SIP or telephony.

The same immediate repeated HTTP 429 errors occurred.

  1. Deepgram Nova-3

To isolate the issue from AssemblyAI, I then tested:

deepgram/nova-3

again directly in LiveKit Test in Console.

Console job:
AJ_rskuetCXwG7G

The same repeated HTTP 429 errors occurred immediately.

This is particularly anomalous because the LiveKit Inference Limits page shows:

  • Deepgram Nova-3: limit 5 concurrent connections, historical peak 3/5
  • AssemblyAI Universal Streaming Multilingual: limit 5, historical peak 2/5
  • AssemblyAI Universal-3.5 Pro Streaming: limit 5, historical peak 0/5

Therefore, a single isolated session is receiving HTTP 429 even though the dashboard shows available concurrency.

The billing/usage page also shows historical successful usage for:

  • Deepgram Nova-3
  • AssemblyAI Universal Streaming Multilingual

but no successful usage for Universal-3.5 Pro.

Could you please verify whether there is:

  • an account-level or project-level LiveKit Inference restriction not exposed in the dashboard;
  • an upstream capacity issue;
  • an unexposed Build-plan concurrency restriction;
  • a mismatch between the concurrency counters shown in the UI and the actual inference gateway state;
  • or another LiveKit Inference gateway issue causing HTTP 429 before the configured per-model concurrency limit is reached?

I can provide:

  • the observability export containing the traceback;
  • screenshots of the Inference Limits page;
  • screenshots of the Console STT errors;
  • additional job/session IDs and timestamps if needed.

The key point is that this is reproducible in LiveKit Test in Console with a single session and multiple STT providers/models, so it does not appear to be caused by my SIP flow or application code.

Thank you.

2

@Giammario_Battaglia Concurrency is only one of the dimensions the inference gateway enforces, which is probably why the dashboard looks inconsistent with what you’re seeing. In livekit/agents, inference/_utils.py defines the gateway’s quota telemetry headers:

  X-LiveKit-Inference-RPM-Limit / -Used
  X-LiveKit-Inference-TPM-Limit / -Used
  X-LiveKit-Inference-Credits-Limit / -Used

with the note that “RPM/TPM are per-minute rate limits; Credits is a cumulative token-credit balance”, and that these are stamped on responses “including 429 rejections”. The Inference Limits page you’re reading reports concurrent connections, so a peak of 0/5 there is entirely consistent with a 429 raised by a per-minute or credit-balance limit instead different counter, not shown on that page.

The credits dimension is the one I’d look at first given your symptom. It’s cumulative and account-wide rather than per-model, which fits a single session failing immediately across Deepgram and AssemblyAI at once. Per-model concurrency wouldn’t produce that pattern.

Unfortunately you can’t confirm which dimension tripped from the client side today, and that’s worth knowing rather than chasing. The STT path drops the evidence:


  except aiohttp.ClientResponseError as e:
      raise create_api_error_from_http(e.message, status=e.status) from e

Only message and status survive e.headers is discarded, so any quota headers on that 429 never reach your logs. The LLM path does the opposite: inference/llm.py calls log_rate_limited() on a 429, which runs extract_quota_usage(e.response.headers) and logs the gateway’s quota snapshot. So the same rejection is diagnosable on LLM and opaque on STT.

Your remaining questions whether there’s an account-level restriction, an upstream capacity issue, or a counter mismatch in the gateway all need someone with access to your project’s gateway state, so I’ll leave those to the LiveKit team rather than speculate.