Agents Regression Testing

Getting Started
, ,
1

Hi everyone!

I’m looking into implementing a regression testing strategy for our self-hosted LiveKit Agents and would love to hear how other teams are approaching this.

As agents become more complex, changes to prompts, LLM models, STT/TTS providers, plugins, tools, and conversation flows can introduce regressions that aren’t always easy to catch with traditional unit or integration tests.

I’m particularly interested in two approaches:

1. Supervised regression testing

Testing agent behavior against predefined scenarios with expected outcomes.

For example:

  • Running predefined conversations with specific user inputs and expected agent responses.

  • Validating tool/function calls, arguments, and execution order.

  • Checking whether the agent follows the expected conversation flow.

  • Evaluating response quality, correctness, and adherence to instructions.

  • Comparing results across different versions of prompts, models, or agent configurations.

Ideally, these tests would run automatically as part of our self-hosted deployment pipeline, before promoting a new agent version to production.

2. Unsupervised regression testing

Monitoring and evaluating real conversations without requiring predefined expected responses for every scenario.

Some ideas I’ve been considering:

  • Replaying production conversations against a new agent version and comparing behavior.

  • Detecting significant changes in conversation patterns, tool usage, or agent decisions.

  • Identifying unexpected conversation endings, interruptions, hallucinations, or failure patterns.

  • Using LLM-as-a-judge to evaluate conversations and flag potential regressions.

  • Comparing evaluation metrics between the current production version and a candidate version.

A few questions

  1. Has anyone implemented a regression testing framework specifically for self-hosted LiveKit Agents?

  2. Are you using tools such as DeepEval, Ragas, LangSmith, Braintrust, or custom evaluation pipelines?

  3. How do you simulate or replay conversations while preserving the behavior of real-time components such as VAD, turn detection, STT, and TTS?

  4. How do you evaluate multi-turn conversations rather than isolated LLM responses?

  5. Do you maintain golden datasets of conversations and continuously expand them using production traffic?

  6. How do you integrate these evaluations into your own CI/CD pipelines?

  7. Are there any LiveKit-specific tools or recommended approaches for running automated end-to-end evaluations without relying on LiveKit Cloud?

I’m particularly interested in self-hosted, reproducible solutions where we have full control over the infrastructure, test datasets, execution environment, and evaluation pipeline.

If you’ve built something similar, I’d really appreciate hearing about your architecture, tooling, datasets, and lessons learned.

Thanks in advance for any help!

2

@Kevin_Quijano, Most of this is first-party now. Three tools cover your two buckets, and all of them run self-hosted with no LiveKit room.

Supervised, turn level: unit tests. AgentSession.run(user_input=...) returns a result you assert on, so you check the exact message, the tool call with its name and arguments, and the order, closing with no_more_events() to catch anything extra. mock_tools stubs a tool, and .judge(llm, intent=...) runs an LLM-as-judge for the qualitative checks like instruction adherence (unit-tests). It runs in pytest or Vitest; point the LLM at a model plugin, as below, and the tests need no LiveKit credentials, because they never open a room.

Multi-turn and flow: Agent Simulations. You write scenarios, group them into scenario sets, and an LLM judges each run, in text mode for logic or audio mode when you want the real STT-LLM-TTS path exercised (simulations), with a CI path for the suite (run in CI). That split answers your real-time question directly: unit tests and text simulations bypass VAD, turn detection, STT and TTS on purpose for determinism, and audio-mode simulations are what put them back in.

Unsupervised, on production: the agent exports OpenTelemetry traces, plus data hooks and audio/video recording, so you capture real conversations and compare a candidate against production in your own stack (tracing). That trace export is also the seam for DeepEval, Ragas or Braintrust, rather than a LiveKit-native binding to any one of them.

from livekit.agents import AgentSession
from livekit.plugins import openai
from agent import Assistant

async def test_weather():
    async with AgentSession(llm=openai.LLM(model="gpt-4o")) as session:
        await session.start(Assistant())
        result = await session.run(user_input="Weather in Tokyo?")
        result.expect.next_event().is_function_call(
            name="lookup_weather", arguments={"location": "Tokyo"})
        await result.expect.next_event().is_message(role="assistant").judge(
            session.llm, intent="States the weather in Tokyo")
        result.expect.no_more_events()
3

For the OTel production seam Muhammad pointed at, Parlot can be useful: goal auto-detection and outcome scoring, a session timeline, quality and speed views on real traffic, plus user-defined judges on the same traces.

Disclaimer: I’m a founder at Parlot. Join the waitlist and DM me: I’ll grant you access.