I was making some tests, and I noticed that an agent that already received a SIGTERM signal (and it’s handling an active call), can accept new requests… is this the expected behavior?. In case it is, is it available any strategy to use to avoid this behavior?
If your container starts the agent through a shell — CMD python agent.py start (shell form), sh -c “…”, or a wrapper script without exec — then the shell is PID 1 and does not forward SIGTERM to Python. Python never sees the signal, never drains, and keeps answering availability as WS_AVAILABLE right up until ECS SIGKILLs it. That’s your “keeps accepting new jobs.”
Make Python PID 1 / forward signals — the real fix. Either:
Use exec form in your Dockerfile: CMD [“python”, “agent.py”, “start”] (not the shell string form), or exec python agent.py start if you use an entrypoint script; or
Set initProcessEnabled: true in the ECS container definition (adds an init/tini as PID 1 that forwards SIGTERM).
Run in production mode (python agent.py start / cli.run_app). Drain is skipped in dev (if not devmode:).
Give drain enough time: set the ECS task stopTimeout ≥ your max call length (default 30s), so an in-progress call can finish draining before SIGKILL. (This won’t affect the “accepting new jobs” symptom — that’s the signal issue — but you’ll want it once drain works.)
About your test
Once SIGTERM actually reaches the agent, the worker reports WS_FULL within a second or two, so by the time you place call_id_2 (10s later) the LiveKit server won’t dispatch it to that worker — no agent joins, and call_id_2 stays “Ring” (your assertion passes), while call_id (the active call) keeps running to completion. That’s the behavior you’re expecting.
Quick thing to confirm so I can be sure it’s the PID-1 case: how does your container launch the agent (Dockerfile CMD/ENTRYPOINT), and do you see the “draining worker” log when you stop the task? If you paste those I can pin it down exactly.
@Kevin_Quijano Your timestamps already answer this, and it isn’t the signal issue.
Call created 14:56:16.524, drain started 14:56:19.638. The call landed three seconds before the worker began draining, so it was still available and accepting it was correct. _is_available() returns False the moment _draining is set, and drain() sets it on the line immediately after that log, so the accepting window closes right there.
The part worth looking at is the 13 seconds between desired_count=0 and the drain log. That API call returning only means ECS accepted the request. The control plane still has to pick the task, deregister it, then deliver SIGTERM. None of that is agent time.
So the test is measuring from the wrong event. Place call_id_2 relative to the “draining worker” line rather than the scaling call and I’d expect your assertion to pass. If you want the actual signal-delivery number, timestamp it inside your own SIGTERM handler and compare against the API call.
One small thing: exec uv run app.py start makes uv PID 1 rather than Python, so there’s an extra hop for the signal. Drain clearly runs so it is forwarding, but worth knowing it’s there if you end up chasing milliseconds.