We run a voice agent on LiveKit Agents (STT→LLM→TTS via the Mistral API; self-hosted LiveKit server), shipped through GitLab CI. Today every MR/PR ends with someone manually placing a call to confirm the agent still answers, turn-taking feels right, and latency hasn’t regressed. I want to automate that.
I’ve seen the first-party test framework and simulations — but both run in text mode, so they cover the LLM/behavioral layer, not the audio path where most of our regressions live (endpointing / EOU-EOT timing, STT final-transcript timing, TTS TTFB/RTF).
For people who’ve automated this:
Is anyone feeding golden audio fixtures through the full pipeline in CI and asserting on the output transcript/intent + latency budgets? How do you keep it non-flaky (tolerance bands, LLM-as-judge on the transcript)?
Any clean pattern for a bot participant that publishes a WAV into a room and captures the agent’s audio track, so it runs unattended?
What do you actually gate a merge on - text-mode behavioral tests only, or a small full-audio E2E suite too? And is the third-party tooling (Bluejay/Cekura/Coval/Hamming) worth it as a per-PR gate vs. a homegrown harness?
Any real-world setups or “here’s what we regretted” welcome.
This is a great question. I think much of your regression detection can be handled with Evals and Simulations that happen with each build.
I think there is a distinction to be made for CI/CD, which tests very often, and the full integration test that may run less often. I think evals and simulations should be run frequently as you work or as code is pushed to your repo.
For full end-to-end testing, I’ve seen folks are happy with some of these (Bluejay/Cekura/Coval/Hamming). I’ve seen many folks in the community happy with Bluejay and Hamming.
In the past, I’ve built my own audio text fixture several times, and it wasn’t all that hard to use SIP or the LiveKit Python SDK to interact directly in a room. You can create another “test” agent that is more static and use it to test your prod agent, too. There is a minor gotcha that, by default, agents can’t talk to each other. You can change the config to enable that. I’ve run this sort of agent over and over, even after the code is live, as a sort of monitoring.
I will be curious to hear what path you end up going and how it works out.
We’ve built pretty much this exact thing internally for our Livekit Dashboard (both a text-mode simulation layer and an audio replay harness) to complete the loop for LLM/STT,
So a few lessons from the scar tissue:
Split the suite into three tiers - don’t gate everything on full audio.
We gate merges on the text-mode behavioral tier (simulated caller LLM + an evaluator LLM scoring the transcript against pass/fail criteria) plus a small audio smoke test - “does the agent join, greet, and
respond to one utterance”. The fuller audio E2E suite runs nightly/pre-release, not per-MR. A per-PR full-audio gate is where flakiness slowly destroys the team’s trust in CI, and once people start clicking
“retry” reflexively you’ve lost the whole value of the gate.
Golden audio fixtures work, but geometry parity is the #1 trap.
Your replay harness must feed audio at the exact sample rate AND frame size the production room input uses - read them from the SDK’s live defaults, don’t hardcode either. We lost a long time to a harness
that replayed at a different geometry than production: it was green for months while being structurally incapable of reproducing a whole class of resampler/frame-geometry bugs that only manifest at the live
geometry. Frame size matters as much as rate - a bug can appear at 50 ms chunks and vanish completely at 10/20 ms. Corollary: pace the replay loop off the actual frame duration, not a hardcoded sleep(0.01), or you’ll replay faster than real time and trip streaming-STT rate limits with fake timing numbers.
Fixture hygiene: caller-side audio only.
If your fixtures come from real call recordings, make sure the agent’s own TTS isn’t in the audio you feed back through STT. Composite (both-legs) recordings give the STT extra acoustic/linguistic context and
your accuracy numbers come out optimistically biased versus live traffic. Mute the agent leg when capturing fixtures.
Non-flakiness on assertions.
Transcripts: normalized WER / similarity thresholds, never exact match.
Intent: LLM-as-judge with a written rubric; judge twice and only fail when both agree (or majority-of-3). Single-shot judges flap.
Latency: don’t assert absolute wall-clock budgets on shared CI runners - the variance swamps the signal. Assert on deltas vs a baseline recorded on the same runner class, and track trends rather than
hard-failing on one slow run.
Better still: the agents SDK emits per-component metrics events (STT final latency, LLM TTFT, TTS TTFB). Logging and asserting on those is far less flaky than mouth-to-ear audio timing, and when something
regresses it points straight at the component instead of “the call felt slow”.
Bot participant pattern.
Plain Python rtc SDK works fine headless in CI: create an AudioSource + LocalAudioTrack, capture_frame() your WAV in 10–20 ms frames paced in real time, subscribe to the agent’s audio track and buffer what
comes back. An energy threshold / VAD on the received frames gives you agent-speech-start timestamps for end-to-end latency. Dispatch the agent to the room the same way production does so you’re testing the real entrypoint.
Third-party vs homegrown.
We went homegrown because we needed exact replay parity with our own pipeline geometry, and that’s the part no external tool can guarantee for you. The hosted tools are worth a trial if you don’t want to own
the harness - but you still own the fixtures, thresholds, and the geometry question either way, and that’s most of the actual work.
Biggest single takeaway: most of our real regressions were in the audio path, exactly as you say, but most of the detection value came from cheap, deterministic component-level checks (metrics events, transcript scoring) rather than the full-audio suite. Keep the full E2E tier small, boring, and rarely run, and it stays trustworthy.