Cc: @nilkanth
The memray flat-RSS chart plus the top output (20+ python3 each at 250-450 MB) shifts the diagnosis away from the worker-level leak candidates identified earlier.
Your session_end_timeout instinct is on track. The default is session_end_timeout: float = 300.0 (“Maximum amount of time to wait for on_session_end to complete (default: 5 minutes)”), as seen in livekit/agents/livekit-agents/livekit/agents/worker.py. If on_session_end blocks or hangs (e.g., network call, unreleased resource, pending task), the JobProcess stays alive for the full 5 minutes per session. Under sustained call volume, processes stack up faster than they exit.
Two things to try:
-
Add an
enter/exitlog inon_session_end(or your equivalent close hook). If the enter log fires but the exit log doesn’t within 5 minutes, that’s the hang. -
Set
session_end_timeout=15inWorkerOptionsand rerun. If process accumulation stops, the close hook is the culprit. If it doesn’t, the leak is inWorkerOptions(thenum_idle_processesprewarm pool, which defaults tomin(ceil(cpu_count()), 4)in production [same file]) or in subprocess termination itself.
Within a single process, RSS is steady. The accumulation is at the subprocess layer: JobProcess instances not being reaped after sessions end.