▎ Deployed agent: all in-cluster inference STT/TTS connects fail with 429 (fresh replica, single session) — public gateway works fine

Agents
, ,
1

Setup

Cloud-deployed agent CA_8bzXYSzJfHAn, project p_5yxrpyr0uln, region ap-south (India West). Build plan, Agents SDK 1.6.4. Voice pipeline entirely on LiveKit Inference: deepgram/nova-3 (STT), cartesia/sonic-3 (TTS), google/gemma-4-31b-it (LLM), inference.TurnDetector().

Problem

Since ~2026-07-05 13:03 UTC, every session dies within seconds of the job starting. The STT websocket connection to the in-cluster gateway is rejected immediately with 429 Too Many Requests; it retries 3× and then the session closes:

failed to recognize speech: message='Invalid response status (429 Too Many Requests)', status_code=429, retryable=True   (attempts 0–2)
AgentSession is closing due to unrecoverable error: ... 429 ...

TTS prewarm fails the same way:

WSServerHandshakeError: 429 on ws://inference.livekit.svc.cluster.local/v1/tts?model=cartesia/sonic-3

Net effect for the user: the agent joins the room and appears alive (participant present, RPCs to the client work), but never hears or says anything.

What I’ve ruled out

  • Load/concurrency on my side: this is a freshly booted replica with exactly one active session and no other rooms (lk room list shows 1). The Build plan limit is 5 concurrent STT connections — I’m well under it.
  • Bad replica / bad version: reproduces after lk agent restart and after a full lk agent deploy (new version).
  • Quota / credit / auth: the same project and same API key connecting to the public gateway (https://agent-gateway.livekit.cloud/v1) works for STT, TTS, and LLM — verified from an external machine with identical models and settings.
  • Platform incident: status page [shows no active incidents — confirm].

Current workaround

Passing base_url="https://agent-gateway.livekit.cloud/v1" to the inference STT/TTS/LLM/TurnDetector in the deployed agent. This restores service but costs a lot of latency — warm LLM TTFT goes from ~510ms (on the public gateway) to ~1.3–1.4s from inside the deployed agent, presumably because the request now leaves the DC instead of using the colocated path. Given [the in-cluster latency figures just published], the in-cluster path is exactly the one I want to be on.

Ask

Can you check what limit the in-cluster gateway believes deployment CA_8bzXYSzJfHAn / project p_5yxrpyr0uln is exceeding as of 2026-07-05 13:03 UTC? Happy to provide job IDs (e.g. AJ_E5BdGUZBLuKc, AJ_YL3WQREfLxtF, AJ_3WH3PyAcssGc) and full log excerpts.

2

Hi, I took a look at a couple of the sessions you listed, AJ_E5BdGUZBLuKc and AJ_YL3WQREfLxtF. Both of those are showing the “STT connection limit exceeded” which refers to Inference concurrency, i.e. this line on the pricing page: https://livekit.com/pricing#inference-concurrency.

Looks like you were running more than the number of concurrent sessions allowed under your plan, which for Build is 5. Note that number is for STT, LLM, and TTS, so a default agent setup would could for 3 of those when running.

I don’t quite understand your workaround, the two options you have are:

  • Upgrade to Ship (The assumption if you are hitting these concurrency limits is you have multiple users using your service) - if you haven’t launched yet, perhaps there is something in your dev / testing setup that is causing a high concurrency.
  • Use plugins instead of LiveKit inference, providing your own keys for your selected model(s)
3

Thanks — concurrency explains part of my history (I did have console/probe tests overlapping phone sessions earlier). But I’ve now reproduced the 429 with a single session on an otherwise idle project (0 rooms before connect, verified via lk room list).

Today, minutes apart:

  • 08:25 UTC — job AJ_KS8hkroJZiRm (only session): STT connected, but the TTS connection-pool prewarm got a 429.
  • 08:29 UTC — job AJ_RawAGhFEMsk2 (only session): STT connect 429 ×3, AgentSession unrecoverable.
  • 08:34 UTC — job AJ_2JRhf4SDFwop, same deployment but with base_url=https://agent-gateway.livekit.cloud/v1: zero 429s.

Questions:

  1. What counts toward the 5? Do the turn detector, adaptive interruption detector, and TTS prewarm connections count? If so, a single default voice agent consumes ~5 by itself.
  2. Are slots released immediately when a session ends?
  3. Why is the same load never rejected via the public gateway — is the limit only enforced in-cluster? (My “workaround” is just pointing the inference base_url at the public gateway instead of the injected in-cluster URL.)
4

The default inference base URL is the value you specified (code is here: agents/livekit-agents/livekit/agents/inference/_utils.py at main · livekit/agents · GitHub). I don’t see why setting that value would have any effect, perhaps it is a red herring.

Quite possibly prewarming TTS is having an effect, I have never tried that. Can you share your AgentSession config?

I would expect slots to be released shortly after the session ends.

5

1. Why setting base_url has an effect on deployed agents

get_default_inference_url() in the _utils.py you linked checks LIVEKIT_INFERENCE_URL first — and the LiveKit Cloud deployment runtime injects LIVEKIT_INFERENCE_URL=http://inference.livekit.svc.cluster.local/v1 into deployed agent containers. So a deployed agent’s default is the in-cluster gateway, not agent-gateway.livekit.cloud. My explicit base_url overrides that injection back to the public gateway — where I never see 429s. Locally the env var isn’t set, so the default there is already the public gateway and setting base_url would change nothing; that’s probably why it looked like a red herring.

2. What it resolves to without my override (deaf-mic condition)

I removed the override entirely (verified via lk agent secrets) and restarted. The agent’s own logs show the resolved URL — agents 1.6.4, region India West.

Session start (job AJ_KS8hkroJZiRm, 2026-07-06 08:25:36 UTC):

{"message": "adaptive interruption", "url": "http://inference.livekit.svc.cluster.local/v1", ...}
[line came through scrambled in source — paste the real one]

TTS connection-pool prewarm failure, showing the websocket target:

aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status',
url='ws://inference.livekit.svc.cluster.local/v1/tts?model=cartesia/sonic-3'

Next real session in that configuration (job AJ_RawAGhFEMsk2, 08:29:20 UTC — the only session in the project; the previous one ended [...] ~2 minutes earlier):

failed to recognize speech: message='Invalid response status (429 Too Many Requests)', status_code=429, retryable=True
AgentSession is closing due to unrecoverable error: ... 429 ...

Restoring base_url=https://agent-gateway.livekit.cloud/v1 on the identical deployment: the next session (job AJ_2JRhf4SDFwop, 08:34 UTC) and everything since — zero 429s. Note that the interruption detector logs the cluster-local URL in [all] sessions (it doesn’t take a base_url from user code), which confirms the env var is injected regardless of what the user passes.

3. AgentSession config

I don’t prewarm TTS myself — the prewarm comes from the SDK’s own ConnectionPool.prewarm in livekit/agents/inference/tts.py at session start. Config (agents 1.6.4):

session = AgentSession(
    stt=inference.STT(
        model="deepgram/nova-3",
        language="en-IN",
        extra_kwargs={"numerals": True},
    ),
    tts=inference.TTS(
        model="cartesia/sonic-3",
        voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
    ),
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector(),        # [truncated in source — confirm]
        preemptive_generation={"enabled": True},
    ),
)

# LLM is on the Agent subclass: inference.LLM(model="google/gemma-4-31b-it")   # [reconstructed — confirm]
# session.start(..., record=False, room_options=RoomOptions(
#     audio_input=AudioInputOptions(
#         noise_cancellation=ai_coustics.audio_enhancement(model=EnhancerModel.QUAIL_VF_S))))
6

I had a closer look this morning and I apologise I didn’t spot this yesterday, but the actual cause of the error is that you have exceeded the gateway credits, i.e. the $2.50 of inference credits you get in the build plan: