We’re seeing consistent silent dispatch failure — rooms are created and callers join, but the agent is never dispatched to them at all, despite the token correctly requesting it. The agent’s status page shows a persistent “CURRENT STATUS: ERROR” badge, but every individual build/version shows “Errors (0)” — so the error isn’t tied to a specific deploy, it looks like an ongoing runtime/health state.
What we can’t see from our side: any runtime/application log for the agent process itself — the dashboard’s log view only shows the Docker build/image-push output, not stdout/stderr from the running Python process. Since the agent never gets dispatched, there’s also no per-call observability data generated for these failures.
Recurring incidents (all times UTC, today, Aug 26): ~08:40, ~09:03, ~14:36, ~15:40 — spanning over 6 hours, multiple separate call attempts, same pattern every time.
Ask: can you check the agent’s health-check/dispatch-eligibility state on your side, and share the actual runtime logs for this agent around these timestamps? We’d like to understand what’s causing the persistent ERROR status and why dispatch requests aren’t reaching a worker at all.
Hi, if I check now I see your agent status as ‘RUNNING’
I don’t have access to your agent dispatch logs I’m afraid.
What we can’t see from our side: any runtime/application log for the agent process itself
You should be able to run lk agent logs and then trigger a dispatch (more info). Alternatively, you could use the agent console to perform an isolated test: Agent Console | LiveKit Documentation
It looks like your last session where an agent was successfully deployed was about 2233 UTC - it might be worth rolling back to an older agent version that corresponded with that dispatch.
Used the Agent Console you suggested — production agent selected, connected for 41 seconds, zero events, agent never joined. same failure
If it was me in your position, my first suspicion was something might be crashing in the agent on launch. If so, that would be shown in the lk agent logs when the agent is deployed.
Darryn, really appreciate your help here. I am a total novice.
Update on the crash loop I was seeing — I found and removed a corrupted sound file that was causing constant av.error.InvalidDataError crashes on every subprocess. That specific error is now completely gone from my logs.
However, a new (or previously masked) issue has surfaced: every subprocess now fails with a plain TimeoutError during proc.initialize(), continuously, for 15+ minutes straight, in tight groups of about four at a time. The worker never successfully registers during this entire window — no "registered worker" line appears anywhere in the logs, even after waiting well past the point where it normally would.
This looks different from the audio-decode issue — no exception, just a timeout waiting for a subprocess to finish initializing. Given it’s been continuous and hasn’t resolved on its own, I wanted to flag it in case it’s a resource/capacity issue on your end rather than something in my code.
Agent ID: CA_QVZmaVJkjShp
Project: Spam Viking
Timeframe: today, roughly 20:42–21:01 UTC (and ongoing when I stopped watching)
Happy to send the full raw logs if useful. Let me know if there’s anything specific you’d like me to check on my side.
Thanks,
Andrew
is the best move here to recreate my agent anew?
No worries. I still can’t see an agent join a session on your project since about 2233 UTC on 25th August.
Whatever changes you made to your agent since then are causing the issue (I assume). Because you have a cloud-hosted agent, you can rollback to a previous agent version - there should be an option under those 3 dots next to the arrow - that will at least get you back up and running and show that something changed, presumably in one of those git commits referenced in the attributes)
That should help you narrow down what the root cause is.
feel free to send me the full logs over DM and I’ll see if I can see anything.
I am slugging it out with claude to try and figure what’s going on.
Llaude found a corrupted sound file (MP3 mislabeled as .wav) that was crashing every agent subprocess on startup with a decode error. That’s gone now. But new issue: every subprocess now times out during initialization (TimeoutError in proc.initialize()) and the worker never successfully registers. This would fully explain your “no join since Aug 25” observation — it’s likely the same underlying blocker, not something new. Rolling back to an older code version probably won’t help here, since the fix was a file-level cleanup, not a code change. Could this be a resource/capacity issue on your platform side?
Thanks for working with me here. Andy
hi darryn, I actually got a direct reply from LiveKit Support (Grant) who’s now looking into this from the platform side. Turns out CPU has been pinned near 100%+ and replicas maxed out since Aug 25, likely tied to a crash-loop bug I found and fixed on my end (a corrupted sound file). He’s investigating further on the infrastructure side now — appreciate your help getting me this far with the CLI/logs approach, that’s what surfaced the real issue.
OK, thanks for letting me know, I’ll leave it to Grant.
