# ▎ Deployed agent: all in-cluster inference STT/TTS connects fail with 429 (fresh replica, single session) — public gateway works fine

**URL:** <https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562>\
**Category:** Agents\
**Tags:** agent-deployment, livekit-inference, livekit-cloud\
**Created:** [July 5, 2026, 5:27pm UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562 "2026-07-05T17:27:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Raqlay\_Labs](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/raqlay_labs/32/1851_2.png) [@Raqlay\_Labs](https://community.livekit.io/u/Raqlay_Labs)\
**Post date:** [July 5, 2026, 5:27pm UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/1 "2026-07-05T17:27:51Z")

</div>

**Setup**

Cloud-deployed agent `CA_8bzXYSzJfHAn`, project `p_5yxrpyr0uln`, region ap-south (India West). Build plan, Agents SDK 1.6.4. Voice pipeline entirely on LiveKit Inference: `deepgram/nova-3` (STT), `cartesia/sonic-3` (TTS), `google/gemma-4-31b-it` (LLM), `inference.TurnDetector()`.

**Problem**

Since ~2026-07-05 13:03 UTC, every session dies within seconds of the job starting. The STT websocket connection to the in-cluster gateway is rejected immediately with 429 Too Many Requests; it retries 3× and then the session closes:

```auto
failed to recognize speech: message='Invalid response status (429 Too Many Requests)', status_code=429, retryable=True (attempts 0–2)
AgentSession is closing due to unrecoverable error: ... 429 ...

```

TTS prewarm fails the same way:

```auto
WSServerHandshakeError: 429 on ws://inference.livekit.svc.cluster.local/v1/tts?model=cartesia/sonic-3

```

Net effect for the user: the agent joins the room and appears alive (participant present, RPCs to the client work), but never hears or says anything.

**What I’ve ruled out**

- **Load/concurrency on my side:** this is a freshly booted replica with exactly one active session and no other rooms (`lk room list` shows 1). The Build plan limit is 5 concurrent STT connections — I’m well under it.
- **Bad replica / bad version:** reproduces after `lk agent restart` and after a full `lk agent deploy` (new version).
- **Quota / credit / auth:** the same project and same API key connecting to the public gateway (`https://agent-gateway.livekit.cloud/v1`) works for STT, TTS, and LLM — verified from an external machine with identical models and settings.
- **Platform incident:** status page `[shows no active incidents — confirm]`.

**Current workaround**

Passing `base_url="https://agent-gateway.livekit.cloud/v1"` to the inference STT/TTS/LLM/TurnDetector in the deployed agent. This restores service but costs a lot of latency — warm LLM TTFT goes from ~510ms (on the public gateway) to ~1.3–1.4s from inside the deployed agent, presumably because the request now leaves the DC instead of using the colocated path. Given `[the in-cluster latency figures just published]`, the in-cluster path is exactly the one I want to be on.

**Ask**

Can you check what limit the in-cluster gateway believes deployment `CA_8bzXYSzJfHAn` / project `p_5yxrpyr0uln` is exceeding as of 2026-07-05 13:03 UTC? Happy to provide job IDs (e.g. `AJ_E5BdGUZBLuKc`, `AJ_YL3WQREfLxtF`, `AJ_3WH3PyAcssGc`) and full log excerpts.

---

<div class="post-metadata">

**Author:** ![darryncampbell](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/darryncampbell/32/20_2.png) [@darryncampbell](https://community.livekit.io/u/darryncampbell)\
**Post date:** [July 6, 2026, 8:10am UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/2 "2026-07-06T08:10:50Z")

</div>

Hi, I took a look at a couple of the sessions you listed, `AJ_E5BdGUZBLuKc` and `AJ_YL3WQREfLxtF`. Both of those are showing the “STT connection limit exceeded” which refers to Inference concurrency, i.e. this line on the pricing page: [https://livekit.com/pricing#inference-concurrency](https://livekit.com/pricing#inference-concurrency).

Looks like you were running more than the number of concurrent sessions allowed under your plan, which for Build is `5`. Note that number is for STT, LLM, and TTS, so a default agent setup would could for `3` of those when running.

I don’t quite understand your workaround, the two options you have are:

- Upgrade to Ship (The assumption if you are hitting these concurrency limits is you have multiple users using your service) - if you haven’t launched yet, perhaps there is something in your dev / testing setup that is causing a high concurrency.
- Use plugins instead of LiveKit inference, providing your own keys for your selected model(s)

---

<div class="post-metadata">

**Author:** ![Raqlay\_Labs](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/raqlay_labs/32/1851_2.png) [@Raqlay\_Labs](https://community.livekit.io/u/Raqlay_Labs)\
**Post date:** [July 6, 2026, 8:38am UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/3 "2026-07-06T08:38:53Z")

</div>

Thanks — concurrency explains part of my history (I did have console/probe tests overlapping phone sessions earlier). But I’ve now reproduced the 429 with a **single session** on an otherwise idle project (0 rooms before connect, verified via `lk room list`).

Today, minutes apart:

- **08:25 UTC** — job `AJ_KS8hkroJZiRm` (only session): STT connected, but the TTS connection-pool prewarm got a 429.
- **08:29 UTC** — job `AJ_RawAGhFEMsk2` (only session): STT connect 429 ×3, AgentSession unrecoverable.
- **08:34 UTC** — job `AJ_2JRhf4SDFwop`, same deployment but with `base_url=https://agent-gateway.livekit.cloud/v1`: zero 429s.

**Questions:**

1. **What counts toward the 5?** Do the turn detector, adaptive interruption detector, and TTS prewarm connections count? If so, a single default voice agent consumes ~5 by itself.
2. **Are slots released immediately when a session ends?**
3. **Why is the same load never rejected via the public gateway** — is the limit only enforced in-cluster? (My “workaround” is just pointing the inference `base_url` at the public gateway instead of the injected in-cluster URL.)

---

<div class="post-metadata">

**Author:** ![darryncampbell](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/darryncampbell/32/20_2.png) [@darryncampbell](https://community.livekit.io/u/darryncampbell)\
**Post date:** [July 6, 2026, 9:27am UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/4 "2026-07-06T09:27:46Z")

</div>

The default inference base URL is the value you specified (code is here: [agents/livekit-agents/livekit/agents/inference/\_utils.py at main · livekit/agents · GitHub](https://github.com/livekit/agents/blob/main/livekit-agents/livekit/agents/inference/_utils.py#L11)). I don’t see why setting that value would have any effect, perhaps it is a red herring.

Quite possibly prewarming TTS is having an effect, I have never tried that. Can you share your AgentSession config?

I would expect slots to be released shortly after the session ends.

---

<div class="post-metadata">

**Author:** ![Raqlay\_Labs](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/raqlay_labs/32/1851_2.png) [@Raqlay\_Labs](https://community.livekit.io/u/Raqlay_Labs)\
**Post date:** [July 6, 2026, 3:14pm UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/5 "2026-07-06T15:14:32Z")

</div>

## 1. Why setting `base_url` has an effect on deployed agents

`get_default_inference_url()` in the `_utils.py` you linked checks `LIVEKIT_INFERENCE_URL` first — and the LiveKit Cloud deployment runtime **injects `LIVEKIT_INFERENCE_URL=http://inference.livekit.svc.cluster.local/v1` into deployed agent containers**. So a deployed agent’s _default_ is the in-cluster gateway, not `agent-gateway.livekit.cloud`. My explicit `base_url` overrides that injection back to the public gateway — where I never see 429s. Locally the env var isn’t set, so the default there is already the public gateway and setting `base_url` would change nothing; that’s probably why it looked like a red herring.

## 2. What it resolves to without my override (deaf-mic condition)

I removed the override entirely (verified via `lk agent secrets`) and restarted. The agent’s own logs show the resolved URL — agents 1.6.4, region India West.

**Session start** (job `AJ_KS8hkroJZiRm`, 2026-07-06 08:25:36 UTC):

```auto
{"message": "adaptive interruption", "url": "http://inference.livekit.svc.cluster.local/v1", ...}
[line came through scrambled in source — paste the real one]

```

**TTS connection-pool prewarm failure** , showing the websocket target:

```auto
aiohttp.client_exceptions.WSServerHandshakeError: 429, message='Invalid response status',
url='ws://inference.livekit.svc.cluster.local/v1/tts?model=cartesia/sonic-3'

```

**Next real session** in that configuration (job `AJ_RawAGhFEMsk2`, 08:29:20 UTC — the only session in the project; the previous one ended `[...]` ~2 minutes earlier):

```auto
failed to recognize speech: message='Invalid response status (429 Too Many Requests)', status_code=429, retryable=True
AgentSession is closing due to unrecoverable error: ... 429 ...

```

Restoring `base_url=https://agent-gateway.livekit.cloud/v1` on the identical deployment: the next session (job `AJ_2JRhf4SDFwop`, 08:34 UTC) and everything since — zero 429s. Note that the interruption detector logs the cluster-local URL in `[all]` sessions (it doesn’t take a `base_url` from user code), which confirms the env var is injected regardless of what the user passes.

## 3. AgentSession config

I don’t prewarm TTS myself — the prewarm comes from the SDK’s own `ConnectionPool.prewarm` in `livekit/agents/inference/tts.py` at session start. Config (agents 1.6.4):

```python
session = AgentSession(
    stt=inference.STT(
        model="deepgram/nova-3",
        language="en-IN",
        extra_kwargs={"numerals": True},
    ),
    tts=inference.TTS(
        model="cartesia/sonic-3",
        voice="9626c31c-bec5-4cca-baa8-f8ba9e84c8bc",
    ),
    turn_handling=TurnHandlingOptions(
        turn_detection=inference.TurnDetector(), # [truncated in source — confirm]
        preemptive_generation={"enabled": True},
    ),
)

# LLM is on the Agent subclass: inference.LLM(model="google/gemma-4-31b-it") # [reconstructed — confirm]
# session.start(..., record=False, room_options=RoomOptions(
# audio_input=AudioInputOptions(
# noise_cancellation=ai_coustics.audio_enhancement(model=EnhancerModel.QUAIL_VF_S))))

```

---

<div class="post-metadata">

**Author:** ![darryncampbell](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/darryncampbell/32/20_2.png) [@darryncampbell](https://community.livekit.io/u/darryncampbell)\
**Post date:** [July 7, 2026, 9:16am UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/6 "2026-07-07T09:16:07Z")

</div>

I had a closer look this morning and I apologise I didn’t spot this yesterday, but the actual cause of the error is that you have exceeded the gateway credits, i.e. the $2.50 of inference credits you get in the build plan:

> **[LiveKit Inference credits - LiveKit Pricing](https://livekit.com/pricing#inference-credits)**
>
> Access LLM, STT, and TTS models with a single API key | Plans designed to scale with your projects

---

<div class="post-metadata">

**Author:** ![system](https://sea1.discourse-cdn.com/flex001/user_avatar/community.livekit.io/system/32/26_2.png) [@system](https://community.livekit.io/u/system)\
**Post date:** [July 22, 2026, 9:16am UTC](https://community.livekit.io/t/deployed-agent-all-in-cluster-inference-stt-tts-connects-fail-with-429-fresh-replica-single-session-public-gateway-works-fine/1562/7 "2026-07-22T09:16:42Z")

</div>

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.