Alerts for when job dispatches fail

Agents
1

Are there any mechanisms through which we can be alerted when job dispatches fail for whatever reason such as due to worker availability issues or other outages? This is currently an monitoring / observability blindspot in our effort to ensure high availability. A logical place to look is webhooks, but I don’t see any events that meet this particular need. Any suggestions to close this gap?

2

Can’t you look at room_started events where there are no agent participant_joined events?

3

@CWilson thanks for the pointer. Will look into that approach.

4

@D3x_Ops were you guys able to fix this issue?

5

CWilson’s room_started-without-agent approach, made concrete: arm a timer on each room_started, cancel it when an agent participant_joined (kind == AGENT) arrives; if it fires first, that dispatch got no worker, so alert.

@app.post("/livekit/webhook")
async def hook(req):
    ev = receiver.receive((await req.body()).decode(), req.headers["Authorization"])
    if ev.event == "room_started":
        pending[ev.room.name] = schedule(alert, ev.room.name, after=10)
    elif ev.event == "participant_joined" and ev.participant.kind == ParticipantInfo.AGENT:
        cancel(pending.pop(ev.room.name, None))      # worker joined, no alert
    elif ev.event == "room_finished":
        cancel(pending.pop(ev.room.name, None))

receiver is a WebhookReceiver; schedule/cancel are your timer of choice. The 10s window is the tuning knob. This catches “no worker ever joined”, which is the failure mode you described; to also catch a worker that joined then died, watch the agent’s participant_left without a room_finished.

6

Yes, we ended up with a solution similar to the pseudo code above. Not a whole lot of options otherwise given current primitives.