Obsolete Cloud workers re-register after SIGTERM and keep receiving new dispatches

Agents
1

Could LiveKit help clear obsolete agent instances while preserving the current healthy deployment, or confirm the supported recovery procedure?

Environment: LiveKit Cloud Build plan, us-east, Python 3.11, livekit-agents 1.8.0, LiveKit CLI 2.13.2.

We found and fixed an application runner bug: after cli.run_app() exited normally during deployment shutdown, our wrapper re-executed the Python process. Logs show an old worker draining, shutting down, and then registering again with the same worker ID. The runner now exits with the CLI status instead. Regression tests verify actual SIGTERM and SIGINT shutdown without a restart.

The corrected production version is Running and its new worker passed an isolated audio check: it spoke first, then returned to listening. However, new explicit dispatches are still sometimes assigned to obsolete workers running previous code. We verified this using the public AgentDispatch job state.worker_id and worker registration logs. CLI restarts have not cleared those old instances.

Timeline on September 11, 2026 (UTC): an obsolete worker received shutdown at 20:48:33 and re-registered at 20:48:38. The latest corrected version deployed at 21:12:16; its worker registered at 21:13:02 and passed audio testing at 21:15. Another new test at approximately the same time still went to an older worker and stayed silent. All isolated test rooms have been deleted.

I understand the documented rolling-deployment grace period is up to one hour. Does that deadline force termination even if an old process re-registers, and is there a supported way to terminate only those obsolete instances sooner? The CLI, published RestartAgentRequest, and dashboard do not expose a force-stop option.

Please preserve the current healthy version. Project, agent, worker, and test-job IDs are available privately if a LiveKit staff member needs them; I have left those and private logs out of this public post.

Docs checked: Deployment management | LiveKit Documentation and Server options | LiveKit Documentation

2

@taf On your third question, there may be a command you’ve missed. lk agent delete takes a --deployment flag, and the docs are explicit:

Omitting --deployment deletes the whole agent. To remove only a non-production deployment and make room for another deployment, always pass --deployment NAME.

So run lk agent versions first to see what’s still registered, then target only the obsolete one. Do not run lk agent delete bare, since without the flag it takes out the whole agent including the healthy version you want to keep.

Also worth upgrading the CLI. You’re on 2.13.2 and current is around 2.18.x, and the agent deployment commands have been moving.

One thing that may explain why restarts haven’t helped: those old processes are running the old wrapper. If they get SIGTERM, the old code re-executes them, so they re-register instead of dying. Your fix only applies to newly deployed processes, so the obsolete ones will keep surviving any signal-based shutdown until something removes them at the deployment level.

Your other two questions need someone with server-side visibility. Worker IDs are assigned by the server (self._id = reg.worker_id in worker.py), so what the grace period does with a re-registering worker isn’t visible from the SDK.