An Agent in 'draining' state can accept new requests?

Hi there.

I was making some tests, and I noticed that an agent that already received a SIGTERM signal (and it’s handling an active call), can accept new requests… is this the expected behavior?. In case it is, is it available any strategy to use to avoid this behavior?

Thanks in advance for any clarification

Agents in a draining state do not accept new calls. Make sure you are sending the SIGTERM to the correct PID.

My agent is running on ECS, what I do is basically stop the task, but it keeps accepting new jobs

This is how my test code looks like:

        async with asterisk.make_temp_call(
            extension = self.EXTENSION_TO_CALL,
            ip = server_main_task.private_ip
        ) as call_id:
            await asyncio.sleep(5)

            agent_main_task.stop(wait = False)
            agent_ecs.update_desired_count(count = 0, wait = False)

            await asyncio.sleep(10)

            async with asterisk.make_temp_call(
                extension = self.EXTENSION_TO_CALL,
                ip = server_main_task.private_ip
            ) as call_id_2:
                await asyncio.sleep(60)
                
                call_id_2_status = await asterisk.get_channel_state(call_id_2)
                logger.info(f"{call_id_2_status=}")
                assert call_id_2_status == "Ring"

            await asyncio.sleep(10) 

            assert (await asterisk.get_channel_state(call_id) == "Up")
            assert agent_main_task.desired_status == "STOPPED" and agent_main_task.last_status == "RUNNING"

            await asyncio.sleep(5)

        await asyncio.sleep(15)
        logger.info("Call ended")

the method ‘asterisk.make_temp_call’ is a contextmanager that creates a new call and ends with when __aexit_ is called.

agent_main_task is the ECS task in which the agent is running and the agent_ecs is the ECS service that handles the Tasks

So on ECS this is almost certainly a signal-delivery problem: SIGTERM never reaches your Python process, so drain() never runs.

Why this happens on ECS

ECS StopTask (and scaling desired-count to 0) sends SIGTERM to PID 1 of the container, then SIGKILL after stopTimeout (default 30s). The agent only drains if that SIGTERM lands on the Python process — the handler is a plain signal.signal(SIGTERM, …):
cli/cli.py#L377-L378 (agents/livekit-agents/livekit/agents/cli/cli.py at e9a3422f47f070fbecfb5e8580d9803f48e1b6d6 · livekit/agents · GitHub)

If your container starts the agent through a shell — CMD python agent.py start (shell form), sh -c “…”, or a wrapper script without exec — then the shell is PID 1 and does not forward SIGTERM to Python. Python never sees the signal, never drains, and keeps answering availability as WS_AVAILABLE right up until ECS SIGKILLs it. That’s your “keeps accepting new jobs.”

The one-line diagnostic

When drain() actually runs, the worker logs this first:
worker.py#L939 (agents/livekit-agents/livekit/agents/worker.py at e9a3422f47f070fbecfb5e8580d9803f48e1b6d6 · livekit/agents · GitHub) — logger.info(“draining worker”, …)

Stop the task and check the agent’s logs:

  • No “draining worker” line → SIGTERM isn’t reaching Python (PID 1 / shell wrapper) or you’re not in start mode. This is the likely case.
  • “draining worker” appears → drain ran; then it’s a timing/routing question instead.

(Recall drain() sets _draining=True → _is_available() returns False → new offers get available=False: worker.py#L1336-L1349 (agents/livekit-agents/livekit/agents/worker.py at e9a3422f47f070fbecfb5e8580d9803f48e1b6d6 · livekit/agents · GitHub).)

Fixes

  1. Make Python PID 1 / forward signals — the real fix. Either:
  • Use exec form in your Dockerfile: CMD [“python”, “agent.py”, “start”] (not the shell string form), or exec python agent.py start if you use an entrypoint script; or
  • Set initProcessEnabled: true in the ECS container definition (adds an init/tini as PID 1 that forwards SIGTERM).
  1. Run in production mode (python agent.py start / cli.run_app). Drain is skipped in dev (if not devmode:).
  2. Give drain enough time: set the ECS task stopTimeout ≥ your max call length (default 30s), so an in-progress call can finish draining before SIGKILL. (This won’t affect the “accepting new jobs” symptom — that’s the signal issue — but you’ll want it once drain works.)

About your test

Once SIGTERM actually reaches the agent, the worker reports WS_FULL within a second or two, so by the time you place call_id_2 (10s later) the LiveKit server won’t dispatch it to that worker — no agent joins, and call_id_2 stays “Ring” (your assertion passes), while call_id (the active call) keeps running to completion. That’s the behavior you’re expecting.

Quick thing to confirm so I can be sure it’s the PID-1 case: how does your container launch the agent (Dockerfile CMD/ENTRYPOINT), and do you see the “draining worker” log when you stop the task? If you paste those I can pin it down exactly.

this is how my entrypoint looks like (entrypoint.sh)

#!/bin/bash
set -eo pipefail

AGENT_ENTRYPOINT=${AGENT_ENTRYPOINT:-'/app/app.py'}
exec uv run "$AGENT_ENTRYPOINT" start --url "$LIVEKIT_URL"

my Dockerfile

CMD ["/app/entrypoint.sh"]

I do see the log “draining worker”, but:

The time when I stopped the task

2026-09-08 14:56:06,518 INFO Updated desired_count=0 of service_name=...
2026-09-08 14:56:16,524 INFO utils.AsteriskService - New call created

The time when the log appeared in the console

2026-09-08 14:56:19.638	[livekit-agent] draining worker

@Kevin_Quijano Your timestamps already answer this, and it isn’t the signal issue.

Call created 14:56:16.524, drain started 14:56:19.638. The call landed three seconds before the worker began draining, so it was still available and accepting it was correct. _is_available() returns False the moment _draining is set, and drain() sets it on the line immediately after that log, so the accepting window closes right there.

The part worth looking at is the 13 seconds between desired_count=0 and the drain log. That API call returning only means ECS accepted the request. The control plane still has to pick the task, deregister it, then deliver SIGTERM. None of that is agent time.

So the test is measuring from the wrong event. Place call_id_2 relative to the “draining worker” line rather than the scaling call and I’d expect your assertion to pass. If you want the actual signal-delivery number, timestamp it inside your own SIGTERM handler and compare against the API call.

One small thing: exec uv run app.py start makes uv PID 1 rather than Python, so there’s an extra hop for the signal. Drain clearly runs so it is forwarding, but worth knowing it’s there if you end up chasing milliseconds.

Okay okay. Got it. Thank you for your help! :smiley: