Any missing plugins on Livekit right now?

Agents
, ,
1

Hi everyone, Laurent from France :waving_hand:

I’ve been digging into the LiveKit plugin ecosystem and want to contribute something. The TTS/STT/LLM lanes seem well-covered, but the surrounding layers (memory, eval, telephony, A/B testing, compliance) maybe not so much?

Would love to hear from people building agents and also the team at Livekit

  1. What are the plugins you wish existed?
  2. Where do you currently work around the gap, rolling your own, gluing two services together, or just living with it?

Will summarize back what I hear.

Thanks

2

LiveKit is technically a module (large addon) so plugins are more integrations rather than true plugins. Yes core lanes (TTS/STT/LLM) are mostly solved. The real pain (and opportunity) is everything around the agent lifecycle. Here’s what people actually end up building themselves over and over and what’s missing, current workarounds people usually use, and what I believe would be beneficial if you’re looking for ideas, just be aware LiveKit is stubborn so you’ll have your work cut out for you.

  1. Persistent + Structured Memory (not just vector DB)

What’s missing

→ Session → user → long-term memory unification
→ Write policies (what gets remembered vs ignored)
→ Conflict resolution (old vs new facts)

What people hack together

→ Redis + Pinecone + custom schemas
→ Ad-hoc “memory summarizer” jobs

Plugin that should exist

Drop-in memory layer with:
→ short_term, session, long_term
→ automatic summarization + pruning
→ policy hooks (PII filtering, retention rules)

  1. Real-time Evaluation / Guardrails (in-stream, not batch)

What’s missing

→ Evaluating agent responses while speaking
→ Interrupting bad outputs mid-stream

Current workaround

→ Post-hoc eval pipelines (too late)
→ Regex / keyword filters (primitive)

Plugin idea

Streaming evaluator:
→ toxicity / hallucination / policy checks
→ can pause, rewrite, or cut audio mid-sentence
→ integrates directly with LiveKit data/audio tracks

  1. Telephony Bridge (done right)

What’s missing

→ Clean abstraction over SIP/Twilio + LiveKit
→ Latency-optimized voice pipelines

What people do

→ Glue Twilio + SIP + WebRTC manually
→ Fight jitter, transcoding, and echo

Plugin idea

“Telephony adapter”:
→ inbound/outbound calls
→ DTMF → structured events
→ built-in jitter buffer tuning for voice agents

  1. A/B Testing + Experimentation for Agents

What’s missing

→ Compare prompts, voices, models in production
→ Measure real conversational outcomes

Current workaround

→ Manual routing logic
→ Logging + spreadsheets

Plugin idea

Experiment layer:
→ split traffic across agents/prompts/models
→ metrics: latency, interruption rate, user sentiment
→ automatic winner selection

  1. Conversation Replay + Debugging (this one hurts)

What’s missing

Full timeline replay:
→ audio
→ transcripts
→ LLM decisions
→ tool calls

What people do

→ Log fragments everywhere and pray

Plugin idea

“Session debugger”:
→ scrub timeline like a video editor
→ inspect each token / audio chunk
→ see why the agent said what it said

  1. Compliance / PII / Redaction Layer

What’s missing

→ Real-time redaction before storage or TTS
→ Regional compliance (GDPR, HIPAA-style patterns)

Current workaround

→ Bolt-on filters after the fact

Plugin idea

Streaming compliance middleware:
→ detect + mask PII in audio + text
→ configurable per region
→ audit logs built-in

  1. Tool Orchestration (for voice-first, not chat-first)

What’s missing

→ Latency-aware tool calling during speech
→ Partial responses while tools execute

Current workaround

→ Standard LLM tool calling (too slow for voice UX)

Plugin idea

Voice-native tool runner:
→ interruptible
→ progressive responses (“Let me check that…” → result)
→ timeout-aware fallbacks

  1. Latency Optimizer / Prewarm Manager

(I literally touched on this problem previously)

What’s missing

Coordinated prewarm:
→ STT
→ TTS
→ LLM
→ audio pipeline

Plugin idea

“Warmup orchestrator”:
→ primes models + audio contexts
→ manages iOS autoplay unlock flows
→ reduces first-token + first-audio delay

  1. Multi-agent Coordination Layer

What’s missing

→ Multiple agents in one room (handoff, roles)

Current workaround

→ Hardcoded routing logic

Plugin idea

Agent router:
→ role-based agents (sales, support, verifier)
→ dynamic handoff mid-conversation
→ shared memory context

  1. Synthetic Load / Voice Testing

What’s missing

→ Simulate 1000 concurrent voice users
→ Stress test audio + agent latency

Current workaround

→ None, or very hacky scripts

Plugin idea

Voice load tester:
→ generates synthetic conversations
→ measures end-to-end latency + dropouts

Highest impact plugins would be to build one of these:

→ (ref #5) (Replay/Debugger) → everyone needs it immediately
→ (ref #2) (Real-time guardrails) → hard + valuable
→ (ref #8) (Latency/prewarm) → directly improves UX (and dev’s sanity)

The meta gap (important)

What’s really missing is a standard plugin interface for:

→ intercepting audio streams
→ intercepting LLM tokens
→ injecting/modifying responses in real time

If you build your plugins around these concepts (middleware-style), it’ll be way more valuable than a single-purpose tool.

Hope this helps.

3

Thanks a lot @_E_r_r_o_n_e_o_u_s_D_e_s_i_g_n_s for the very detailed answer! this is very helpful.

First, if I understand well, there is actually a missing middleware piece that would let you compose interception/observation around the existing plugins instead of replacing them? And all of those ideas would only work if there was this clean interface?

Also if you have one more minute I am curious to know:

  1. On prewarm. Is it mostly cold start (first call after idle) or per session warmup of STT/TTS/LLM? or both?
  2. On replay/debugger. It seems like there is a missing PostHog. Did you also try to hack this one yourself? if yes, what was the hardest parts?
  3. You mentioned LiveKit can be stubborn. What did you mean by this? :eyes:

Thanks a lot

4

It is interesting to see that so many of the items above are already available; we may not be doing a good job of communicating these.

For the timeline replay, have you tried Agent Insights?