I put LiveKit on an ESP32-S3 + ReSpeaker XVF3800 — a full-duplex smart speaker that listens while it talks

Robotics
, ,
1

Hi all :waving_hand:

I’ve been building Sebastian, a real, standalone smart speaker you can put on your desk — a hands-free voice assistant that runs LiveKit’s WebRTC stack on a bare ESP32-S3 microcontroller (not a Pi, not a phone — a $7 MCU with 512KB SRAM + PSRAM). You say a wake word, talk to it, and it talks back through its own loudspeaker. It talks to a LiveKit Cloud room, and a Python `agents` worker drives the conversation with a speech-to-speech realtime model.

Most LiveKit projects I see are web/mobile/server. Running the `client-sdk-esp32` on real embedded hardware turned up a bunch of LiveKit-specific patterns and gotchas that I think are worth sharing — some clever, some painful. Would love feedback from anyone doing embedded or agents work.

The stack

  • Device: XIAO ESP32-S3 + a ReSpeaker XVF3800 (XMOS) doing beamforming/AEC on-chip. Firmware in Zig over ESP-IDF, using `client-sdk-esp32` (Developer Preview).

  • Transport: LiveKit Cloud (WebRTC, Opus, SRTP).

  • Agent: Python `livekit-agents` ~1.6, a `RealtimeModel` (Gemini Live / OpenAI Realtime, swappable), + `noise-cancellation` (BVC).

  • Wake word runs on-device (microWakeWord); a session only opens after the wake word, so it’s idle/cheap the rest of the time.

The LiveKit bits I found most interesting

1. Pre-roll over a data byte-stream, not the media track

The hardest UX problem: when you say the wake word, the first ~2 seconds of your command happen before the room is even connected. You can’t inject that audio into the media track after the fact (pts/jitter/ordering make it hopeless).

Solution that worked great: the device records a 12s ring buffer in PSRAM, and at the mic hand-off it ships the pre-roll as a framed byte stream over a LiveKit topic (`register_byte_stream_handler` on the agent side). The agent parses it, resamples it, and injects those frames into a custom `AudioInput` ahead of the live audio. Nothing is lost, and the model hears the full utterance including the part before “connected”.

2. Custom `agents.io.AudioInput`

Instead of the agent subscribing to the track directly, I feed the session a custom `AudioInput` subclass. It merges the pre-roll frames + the live track, exposes a `preroll_ready` event so the entrypoint can skip the greeting, and owns its own resampling. This turned out to be a really clean extension point for embedded, where the audio timeline isn’t “just a track”.

3. Explicit agent dispatch via the token — and a gotcha

The device does a plain HTTP GET to a tiny token server that mints a short-lived JWT with `RoomConfiguration(agents=[RoomAgentDispatch(agent_name=…)])` embedded. So the named agent auto-dispatches when the device joins — no separate dispatch call.

Gotcha that cost me hours: `RoomConfiguration` in the token only applies when the room is created. On a re-wake into an existing room, the dispatch is silently ignored → “the agent never joins the second time”. Minting per-session + letting the room close between sessions fixed it.

4. RealtimeModel provider factory + a real-world fallback lesson

I use a provider factory so `SEBASTIAN_MODEL_PROVIDER` swaps between `google.realtime.RealtimeModel` and `openai.realtime.RealtimeModel`, with turn detection owned by the model (`TurnHandlingOptions(turn_detection=“realtime_llm”)`).

Two lessons:

  • The Gemini native-audio preview model rejected explicit BCP-47 language codes (`es-ES` → `APIError 1007`, session closes, agent goes silent while the device session stays open). Fix: let it auto-detect.

  • It also throws transient `1011 internal error`s. Having the OpenAI fallback wired as one env var saved me — highly recommend the factory pattern if you’re on a preview model.

5. Background Voice Cancellation for a TV in the room

`noise_cancellation.BVC()` on the incoming track is the reason a TV playing in the background doesn’t get transcribed as user speech. Plain NS won’t strip voices BVC does. Combined with the on-device beamforming, background chatter basically disappears.

6. Barge-in via a data channel

Wake-word-over-agent-speech = interrupt. The device publishes a `sebastian.barge_in` data packet; the agent interrupts the current generation. A second `agent_state` data topic mirrors speaking/listening so the firmware knows when to gate the mic (half-duplex fallback).

7. Playback: Opus off the wire, straight to the speaker

The output side matters too — it is a speaker after all. The agent’s TTS comes back as an Opus track over WebRTC; the device decodes it on the MCU and plays it through its loudspeaker via `av_render`. Two things fell out of this nicely:

  • The speaker’s render-reference callback (the exact PCM going to the driver) doubles as my “is the agent talking?” signal — the data-channel-independent keepalive I mention below. No need to trust the data channel to know when audio is flowing.

  • That same playback is fed to the XVF3800 as the AEC far-end reference, so the mic array cancels the speaker’s own voice. The device can listen while it speaks — you can interrupt it mid-sentence and it hears you over its own output.

Two things that aren’t really LiveKit — but people keep asking

It listens while it talks (on-device AEC → real full-duplex)

The one I’m proudest of: it’s genuinely full-duplex. The XVF3800 takes the speaker’s playback as the AEC far-end reference and cancels the device’s own voice out of the 4-mic array — so you can interrupt it mid-sentence and it hears you over its own output, on a $7 MCU. Two things I learned the hard way:

  • Getting the canceller to converge meant freezing the beamformer to a fixed direction. An adaptive beam that tracks the talker is a non-stationary echo path, and the AEC can never lock onto it — it just chases a moving target forever. Fixed beam → converges in ~1s.
  • A neat side effect: because the TV in the room is off the beam axis, it’s attenuated before the wake word even sees it. Background chatter mostly never crosses the “someone is speaking” threshold on-device — the cloud BVC is a second line of defense, not the first.

The firmware is Zig, not C/C++

The whole device side is Zig on top of ESP-IDF — I wanted memory safety and compile-time config without dragging in a heavier runtime. What made it work:

  • The ESP32-S3 is Xtensa, which isn’t a mainline LLVM target, so I build against Espressif’s Zig fork (`kassane/zig-espressif-bootstrap`) wired into the ESP-IDF cmake build.

  • Instead of translating the IDF headers with Zig’s `cImport` builtin (it chokes on the giant structs), I hand-declare only the `extern fn`s I use, plus thin C shims where the C API is more ergonomic.

  • `comptime` handles per-install config (which mic slot, fixed beam, etc.) at zero runtime cost.

  • Footgun for anyone trying this: Zig’s `@`-builtins like `min` **narrow the result type** to a `comptime_int` argument’s range — it bit me with five identical overflow crashes before I spotted it.

The painful one: an SCTP storm on the ESP32

This is the one I’d most love input on. Mid-session, the device’s `esp_peer` (the vendored WebRTC core under `client-sdk-esp32`) starts flooding `SCTP: Send INIT chunk` hundreds of times, never associates, and ends in `SCTP_ABORT` — while the media (SRTP) keeps working fine on its separate transport. So audio survives, but the data channel dies mid-conversation.

Because I was (initially) relying on the data channel to keep the session alive, sessions got cut off. Two things helped:

If anyone here has run `client-sdk-esp32` in production and tamed the SCTP/data-channel reconnect behavior, I’d love to compare notes.

Observability

The agent exports OTLP (traces/metrics/logs) and I stream device telemetry (serial → OTLP) into the same Grafana stack, including per-turn transcripts. Debugging “why didn’t it respond” became “read what it actually heard” — which is how I found both the language-code and SCTP issues. Highly recommend instrumenting the agent early.

Try it / links

Honestly, this project is bigger than me

I’ll be upfront: I stitched this together well past my comfort zone. Embedded WebRTC, real-time DSP, and agent orchestration are not my home turf, and I’m sure there are things here I’ve done naively, worked around when I should have fixed them properly, or just plain gotten wrong. That’s exactly why I’m posting it rather than quietly shipping it.

So what I’d genuinely value is honest, critical feedback — tell me what you’d have done differently, where the sharp edges are, what I’ve misunderstood about LiveKit, WebRTC on constrained devices, or the agents pipeline. Be blunt; I’d much rather learn than be reassured.

Happy to go deep on any part (the wake→dispatch→pre-roll handoff, the token server, the custom `AudioInput`). Thanks for reading — and thanks for the docs and examples that got me this far :folded_hands:

2

Hey @Ruben_Garcia I loved the idea! Out of a pure curiosity - what’s the main purpose btw? Is it just fun educational project, or do you have any particular problem in mind or maybe even starting a business around it?

3

Good job figuring this out. Yes, if your board has the hardware to do it, beam-forming in the mic array is the best way to handle full-duplex operation, particularly in noisy environments. Having multiple mics that are not directly aligned with the device’s audio output is also an important part of this. This will likely help more than BVC does.

…If anyone here has run `client-sdk-esp32` in production and tamed the SCTP/data-channel reconnect behavior, I’d love to compare notes…

In projects I have worked on, I do not keep the ESP32 client connected all the time. It is more like PTT but with a wake word. Once idle for some amount of time it will timeout and transition back to offline until the wake word is spoken again. The issue is how to handle the audio that is spoken between the wake word and the room-connected event.

You are inspiring me to finish up this LiveKit ESP32 series I was working on, which tries to help with several of the items you outlined in your post.

What are you using for your keyword spotter? While working on the blog series, I built a WakeNet (wn9) tool to play with ESP32 wake word models in your browser. Not sure if you will find it helpful at all.

4

I just wanted to have my own smart speaker at home. I love LiveKit and was looking for a personal project where I could use it, so it was mostly a personal challenge.

5

Good call on the mic geometry — though in my case the speaker’s already external by default (the board has no built-in speaker, just a jack/amp-out), so I’m curious how much that alignment point buys you when the speaker’s not baked into the same enclosure to begin with.

Fun find on my end: routing through the XVF’s comms channel (non-linear suppressor) instead of the raw ASR beam killed the echo even with the beam still adapting (unconverged AEC) — that’s from a lab probe though, not shipped yet, still need to wire it in and see how it holds up with real double-talk.

SCTP storm is very real here too, matches esp-webrtc-solution#186. Not fixed it, just contained it — I keep a heartbeat outside the data channel so a dead session doesn’t look alive, and alert on SCTP re-init counts. If you’ve found anything better than “detect and reconnect” I’m all ears.

Same PTT+timeout+wake pattern here. Solved the audio-gap problem with a rolling buffer in PSRAM — on wake I grab the last ~2s before it plus everything through connect, so the first sentence arrives whole, no waiting around. Can dig up details if it’s useful for the series.

Went with microWakeWord over WakeNet mainly because WakeNet doesn’t do custom Spanish out of the box. Currently fighting false positives though, so that browser tool sounds handy — does it let you plug in custom/non-English keywords, or is it tied to the stock models?

6

Yes, WN9 does not have custom words. Curious what Spanish word you were wanting.

The browser tools you can use (upload) any WN9-compatible model. I created a custom one for “Hey LiveKit” that I was using.

7

Sebastian, is a Spanish name and the name of the repo

8

I indexed the documentation with deepwiki so it will be easy to check it

9

Hi @Ruben_Garcia , I would like to address the above issue you had so future developers don’t have the same trouble. I notice we do document this at Agent dispatch | LiveKit Documentation but of course nobody is expected to read all the documentation! I wondered how you developed your dispatch logic, and how you eventually worked out the solution?

10

Hi! Thanks for picking this up.

How the dispatch logic evolved. The client is a bare ESP32-S3, so the only ceremony it can afford before joining is a single HTTP GET to a tiny token server. That constraint drove everything:

  1. First prototype: agents dev mode + a sandbox token — automatic dispatch. Even then it effectively only worked on freshly created rooms, so my “workflow” between tests was literally delete the room and reset the board.
  2. Moving to a named agent (agent_name), token-embedded RoomConfiguration(agents=[RoomAgentDispatch(…)]) looked like the natural fit for embedded: the device stays dumb, and all dispatch knowledge lives server-side in the token mint.

How the gotcha showed up and how I worked it out. The symptom was: answers the first time, says nothing after a quick re-wake. Three log sources triangulated it:

  • device serial: room Connected, but no agent participant ever appears;
  • the worker logs: a JT_ROOM job on the first wake, no job request at all on the re-wake;
  • room state on the server: the room from the previous session was still alive (empty rooms linger for the departure timeout).

The correlation was that it only failed when re-waking before the old room got cleaned up. That narrowed it to “dispatch must be tied to room creation” — and only then did I find the line in the docs confirming it. So yes, it’s documented; the problem is the failure is silent, so nothing in any log points you back to that page.

Update — I’ve now adopted the design your docs actually recommend. Re-reading the page you linked with fresh eyes made me realize my post-gotcha fix (token config + an explicit-dispatch fallback) was two mechanisms doing one job. I’ve since simplified the token server to explicit-only dispatch: plain join token, and create_dispatch (which creates the room if needed) awaited before the token is returned. The whole “does the room already exist?” failure class is gone by construction, and one nice property fell out: if dispatch fails, the server returns 503 and the device fails fast, instead of joining an agentless room and dying by silence timeout.

Two pieces of feedback that would helped me:

  1. A server-side WARN when a join token carries RoomConfiguration.agents but the room already exists and the config is being ignored. The information exists at exactly the moment the mistake happens — today it’s dropped silently. (A callout box on the dispatch page would help too, but the log line is the discoverable one.)
  2. Worth documenting on that page: create_dispatch is not idempotent — dispatching into a room that already has the agent adds a second agent, and the two talk over each other. The guard you need is “check the room for an existing AGENT-kind participant first”. I suspect anyone moving from token-based to explicit dispatch will hit this.

Happy to share the token server (~100 lines of Python) if it’s useful as a docs example.

11

One thing worth noting here again is that we generally recommend against reusing rooms when agents are involved. If you can use a unique room name each time you side step, all that extra complexity. That is what is mentioned in the doc that Darryn shared in this section:

Also, for your second item, I don’t believe it to be accurate:

Worth documenting on that page: create_dispatch is not idempotent — dispatching into a room that already has the agent adds a second agent, and the two talk over each other. The guard you need is: “Check the room for an existing AGENT-kind participant first.” I suspect anyone moving from token-based to explicit dispatch will hit this.

  • Agents do not talk to each other by default; you need to change their config to enable that ability.

  • You should only see two agents join the room if they have different names or a default agent (no agent_name) and a named agent.

We strongly recommend only using named agents.

12

Thanks — both points landed, and I acted on the first one immediately.

On unique room names: adopted, and you were right about the payoff. My token server now mints sebastian- per session and explicitly dispatches into it. The diff was deeply satisfying: the AGENT-participant guard, a “stale agent-only room” self-heal I’d written earlier the same day, and the whole “does the room already exist?” reasoning — all deleted. Fresh room per wake = no shared state to defend. For anyone else with an embedded/wake-word device: the room name lives entirely server-side (the device just uses whatever token it’s handed), so this costs nothing on the client.

On my “agents talk over each other” claim — let me sharpen it, because you’re right about one half and I have server logs for the other half. You’re correct that agents don’t hear each other by default (they don’t subscribe to agent participants), so there’s no agent↔agent conversation loop — I worded that badly. What I observed (and can reproduce on self-hosted v1.13.3) is different: two create_dispatch calls for the same named agent into the same room create two independent jobs → two instances of that agent in the room. They don’t talk to each other — they both answer the user, simultaneously, so the speaker plays two overlapping TTS streams. From the couch that sounds like “it’s talking over itself”. With reused room names, a dispatch race (e.g. a token-fetch retry) makes this reachable in practice; with unique rooms per session, it’s structurally impossible — one more reason for your recommendation.

13

Thanks for the info. I will look into it. Do you have a room session ID where that happened? I would like to check our server logs.

14

I didn’t capture the room SID client-side, but here’s everything to locate it: project demo-ja18uvd7.livekit.cloud, room name sebastian, device participant esp32-respeaker (ESP32 SDK 0.3.10). The cleanest documented repro on Cloud was 2026-07-08, ~13:27–13:35 UTC — agent worker AW_yVHpNjFVSVif, job AJ_vXrHGBXHAAn2. During that session my device-side serial shows 232 SCTP INIT retransmits (~9/s, no backoff) on one association while the other established fine.

One heads-up before you dig: later that same day I migrated to self-hosted OSS v1.13.3, and it reproduces identically there — the publisher transport’s DTLS never completes (your server logs dtls timeout: read/write timeout on the data channels at participant close) while SRTP media stays healthy on both. So it doesn’t look Cloud-specific; it behaves like a client-side SDK issue. Happy to share the self-hosted server logs too if they’re useful for comparison — being able to see both sides is how I got this far.

15

Looking here I only see one agent and one dispatch for that room:

https://cloud.livekit.io/projects/p_/sessions/RM_xyjwzN7pMBgB

16

okay, I will rewire to livekit cloud and I will told you when I would have enough tests reported

17

Today I made a great improvement, Sebastian no longer have to work with the server in my laptop. I build a GitOps system using please-release, kubernetes, helm and argocd to deploy it at my homelab in a server.

All this code is in the repository

image
image1920×1076 165 KB