Was it us or them? Answering "the live froze for me" with evidence

Client SDKs
,
1

When a viewer says “the live froze for me”, someone has to answer a hard question: was it us, or them?

Most live-video teams can’t answer it quickly. Server logs look clean because the servers did their job, dashboards show room averages, and by the time the ticket arrives the moment is gone. It usually ends with “probably your Wi-Fi”.

What works for us is to stop looking at that one viewer in isolation. We reconstruct what every participant actually received and compare the person with everyone else watching the same streams at the same moment. If everyone degraded together, look upstream. If only one did, look at their network and their device.

Two real, anonymised cases from the write-up. In one room the average showed about 1% of video lost, but one viewer was losing around 9% for much of the session: her connection, not the platform. In another, the host’s voice broke up for 36 of 37 viewers at once while a co-host was fine: the host’s upload had hit a bandwidth limit, not the server.

To picture how it works in practice, imagine a product launch streamed to 500 viewers. At 20:14 the CEO’s audio drops out, and support gets twenty messages within a minute. You type in one of them: “Anna, around 20:14”. The answer comes back in one sentence: not her fault, 430 of the 500 lost the host’s audio at the same moment, and the host’s laptop had hit its upload limit a few seconds earlier. Then you scrub the reconstructed scene to 20:14 and see it for yourself: the host’s stream thins out, the whole audience goes quiet, and one viewer in the corner was already struggling on an overheating phone before any of it started. Twenty tickets get one honest answer, and the fix goes to the right place: the host’s connection, not your servers.

Disclosure: we’re building Rewitness. A small SDK records what each participant experienced, and a deterministic rules engine lines those witnesses up on one timeline and says whose side the problem was on and what exactly happened, with the evidence. You can also scrub the reconstructed scene to any moment. It infers server-side problems from what clients received (it doesn’t read your SFU’s logs) and only sees apps that include the SDK.

How do you handle “it was broken for me” reports? Do you collect client-side stats from every participant, or only when someone complains?

4

That’s the right approach.

Assuming you are using LiveKit cloud, you can look at the session dashboard and see stats about each published and subscribed track for each participant.

5

Thanks Darryn, good to hear your perspective. The session dashboard is where I’d start too.

The part that became slow for us was comparing participants. To answer something like “was it only this viewer?”, I had to go through the other viewers one by one and check whether they had issues with the same track at the same time. With 30+ viewers, that quickly becomes most of the investigation.

And with self-hosted rooms, you don’t have that dashboard, so you’re back to collecting and analysing getStats() yourself.

Is there any server-side signal you’d trust to confidently rule the SFU in or out in that case?

6

Honestly, these kind of issues are almost always connection issues with participants.

Not a solution for everyone, but Scale plans also have access to the Analytics API, Analytics API | LiveKit Documentation , which can provide some of this information.