After upgrading to 1.6.4 from 1.5.8, getting: "job process exceeded memory limit, killing it"

Agents
,
1

After upgrading livekit-agents from 1.5.8 → 1.6.4, our job processes started getting killed by the worker’s memory monitor with:

job process exceeded memory limit, killing it

This started appearing immediately after the upgrade was deployed (zero occurrences before it) and is recurring across every build since.
Memory grows steadily over the life of a single call until it crosses the limit.

Environment

  • livekit-agents==1.6.4
  • Python 3.12, Linux, running in Kubernetes
  • Voice pipeline: STT → LLM → TTS, with Silero VAD (prewarmed per process) and a turn detector

Symptom / data from a killed job

memory_limit_mb: 1024
memory_warn_mb: 700
baseline_memory_mb: 564.5
growth_memory_mb: 479.9 ← grows over the call
memory_usage_mb: 1044.3 ← killed here
uptime: ~3800s

So the per-job process climbs ~480 MB above baseline over a single (long-running web) call and gets killed.

Worker config (AgentServer)

AgentServer(
…,
job_memory_warn_mb=700,
job_memory_limit_mb=1024,
load_threshold=0.9,
num_idle_processes=N,
)

These limits are unchanged from before the upgrade — memory usage rose to meet them, the ceiling didn’t drop.

What we’ve checked

  • The memory-limit settings have been the same for a long time; only the livekit-agents version changed.
  • The growth is per job process, not the worker or the shared inference process.
  • This is reproducible on long calls and tracks with call duration.

Questions

  1. Are there any known per-job memory regressions / leaks in 1.6.4 vs 1.5.8 (session, audio I/O, chat context, or the inference path)?
  2. Is the growth expected behavior that we should accommodate by raising job_memory_limit_mb, or does it indicate something not being released
    per turn?
  3. Any recommended way to profile a single job process in 1.6.x to pinpoint where the growth accumulates?

Happy to share more logs or a minimal repro. Thanks!

2

Hi, no, there are no specific known issues. The two most recent ones which come to mind are: https://community.livekit.io/t/upgrading-python-livekit-agents-to-1-5-6-causing-memory-issues/976/17 which was not-reproducible and resolved by moving platforms, and https://community.livekit.io/t/memory-leak-in-google-stt-when-no-audio-input-exists/1023 which was resolved by a merged GH PR. Note these were with 1.5.6, and don’t cover the versions you are reporting.

Some growth would be expected for long-running conversations, but not 480MB, that would not be expected.

If there is a minimal repro you could share, that would be great. Also, how are you hosting your agents?

3

Hi Darryn,

Thanks, I found that reproducing the steps will be difficult. Are you expecting some specific logs?
What we found, we didn’t push only the upgrade causing that problem. That means some part of the update mightn’t directly align with our codebase. we will digdown deeper and get back to you.

4

As I’m sure you can appreciate, these kind of memory issues are tough to pin down and root cause. Ideally, knowing the exact version where the issue was introduced would help, as would knowing the exact setup.

I haven’t seen any other reports similar to this since the two I provided above (either in this community, or reported to our support team) but I’ll keep an eye out.

5

Yeah i am also facing same issue of memory increase just by increasing the versions as i mentioned here

The memory usage has increased more than 1gb for version 1.6.4 now as i can see warn logs again. so on every version upgrade the memory is increased, despite no changes in code

6

I think memory usage would be fixed soon? as mentioned here

7

Thanks for sharing @Kaushal_Shah , from that PR:

I believe this is going to be addressed by #6082. On Linux/forkserver, the memory usage reported double counts parent process memory usage, even if it is from copy-on-write pages. The actual usage should stay the same. Previous turn-detector plugin doesn’t have this issue because it uses it’s own process to isolate the memory, and in the new version, we leverage copy-on-write to save the memory and drop the process requirement.

So, this is a reporting issue on the job memory, which will be addressed in #6082, and in the mean time can be worked around by increasing the job_memory_limit_mb.

8

I am no python expert, but have been actively using this library. I am still confused why a process for a single voice ai call can take up to 500MB of memory. I would love to have a technical detail on what consumes so much memory for this voice ai functions. I know there are audio buffers, etc. but they all are consumed reaaltime, so even if there is memory usage, it cannot be 500 MB. Please excuse my limited understanding of internal systems and processes.

9

Great question — the honest answer is that almost none of it is audio. Audio frames are tiny (a 10ms frame of 48kHz mono PCM is ~1KB), and as you said they’re consumed in realtime, so buffers account for single-digit MB at most. The memory is mostly the stack the call runs on, not the call itself.

Here’s roughly what lives inside a job process:

1. A full Python interpreter plus every imported library. Each call runs in its own OS process, for isolation (one misbehaving call can’t take down the others) and to avoid GIL contention between calls. That process carries the entire import graph: livekit-agents itself, the Rust-based livekit RTC SDK (a complete WebRTC stack — codecs, jitter buffers, resamplers), aiohttp/grpc, numpy, onnxruntime for Silero VAD, plus whichever plugin SDKs you use (openai, deepgram, cartesia, …). Just importing this stack typically costs 150–300MB before the call even starts. This is by far the biggest chunk.

2. Model weights and inference sessions. Whatever you load in prewarm (e.g. the Silero VAD ONNX session) lives in the process’s address space. The heavier turn-detector model runs in a separate shared inference process, so its cost is amortized across all concurrent calls rather than paid per call.

3. Allocator behavior. CPython + glibc malloc rarely return freed memory to the OS, so a process’s RSS behaves like a high-water mark. A transient spike (a large LLM response, a TTS synthesis burst) stays visible in RSS even after the memory is logically freed.

4. And the key point for this thread: the reported number was inflated. The worker forks job processes from a parent that has already imported everything. With copy-on-write, the child shares those library pages with the parent — they exist once in physical RAM. But RSS counts shared pages fully in both the parent and every child, so the per-job figure the framework logged was double-counting memory that isn’t actually duplicated. That’s the reporting bug fixed in #6082, which switches measurement to PSS (each process is charged only its proportional share of shared pages).

A picture of the process model:

flowchart TB
    subgraph worker["Agent worker (one per machine)"]
        parent["Parent process<br/>imports agents + WebRTC (Rust) + onnxruntime + plugin SDKs<br/>≈150–300MB of library pages, loaded once"]
        inf["Shared inference process<br/>(turn detector model)<br/>cost amortized across all calls"]
        j1["Job process — call A<br/>per-call state: audio buffers,<br/>chat context, transcripts (small)"]
        j2["Job process — call B<br/>per-call state (small)"]
    end
    parent -- "fork, copy-on-write:<br/>library pages shared, not copied" --> j1
    parent -- "fork, copy-on-write" --> j2
    j1 <-. "VAD / turn detection RPC" .-> inf
    j2 <-. "VAD / turn detection RPC" .-> inf

So the mental model is: a few hundred MB is the fixed cost of the Python + WebRTC + inference tooling stack, and after CoW it’s mostly shared across every call on the machine. The truly per-call state — audio buffers, the growing chat context, transcripts — is comparatively tiny. RSS just made it look like every call paid the fixed cost in full.

Practical takeaways: upgrade to a release that includes #6082 so the reported numbers reflect reality (and the limit applies to real usage); do heavy loading in prewarm so it happens once in the parent and gets shared; and keep heavyweight imports (like torch) out of your entrypoint unless you actually use them.

10

@Sagor_Sarker, Until it lands, the per-job figure is inflated by RSS counting the parent’s copy-on-write pages (as described above), not a per-turn leak, so the interim is just to raise the ceiling to fit the over-counted number:

AgentServer(..., job_memory_warn_mb=1400, job_memory_limit_mb=2048)