On iOS I’ve been struggling with this situation where immediately on session enter, we output an agent greeting message, and this message echoes into the microphone. It often actually gets transcribed (but mangled), so the agent then fires off a reply to itself. After a few seconds, AEC kicks in and the problem goes away.
Problem is also present on audio re-acquire, for example, if we start playing music on the device on apple music, stop the music, foreground the session, and unmute, the agent’s next turn will echo once again.
The reason I say we’re stuck between a rock and a hard place: there is no easy way to simply mute the user on entry, as this either breaks entirely or causes AEC to not even kick in, leaving the only available path forward to be: don’t connect audio until user first unmutes, then do a full connect. However this means that the first unmute takes a few seconds. Very suboptimal. Not sure what to do here.
I saw a few old issues on GH relating to mixWithOthers, but no fixes were applied.
You should be able to reproduce this bug very easily: just run livekit’s template swiftui voice app, turn up your volume, connect to a session on iOS, and if the agent is configured to greet, it should echo back.
original issue, resolution clearly not fully reached: AEC (Echo Cancellation) regression in v2.12.0 — agent hears its own audio output as user speech · Issue #916 · livekit/client-sdk-swift · GitHub
I see your comment on the GH issue saying this is still not fixed, did you try with 2.12.1 out of interest, to see if it’s a regression.
That’s the right place to fix this, but I do wonder, as a workaround, if it is possible to set a high endpointing delay for the first turn, then reduce it once the AEC settles, Turn handling options | LiveKit Documentation (I haven’t tried it, but it’s an alternative to muting tracks which you already tried)
I just tried it with 2.12.1, AEC issue is still present. I then downgraded to 2.11.0, and it’s still present.
If I were to crank up min endpointing delay for the first turn, we’d still prepend some agent self-speech to the user turn, would we not?
I was just thinking out loud, I would rather solve the actual problem.
Looking back at the GitHub issue, I do notice the comment immediately prior to yours,
Yes, still happening with a clean build. If it helps, it seems to only be happening at the beginning, maybe 2 to 3 rounds of conversation. Then it becomes ok.
which exactly describes your problem.
I also note the following on the original PR:
Pinned client-sdk-swift to 2.11.0 — echo immediately gone, works perfectly
So, I assume if you also pin to 2.11.0 you won’t see this issue either (also the comment 3 above yours on the issue says it fixes it for them)
I think the best thing would be if you could please raise a new PR, with clear repro steps, I will bring it to the attention of the client team. To me, it feels like this is a separate issue since they could not repro with 2.12.1.