Terrible times with vapi + dialpads. Is livekit going to be any better

Egress / Ingress
,
1

We want to build outbound voice agents (using number A) that can handle dialpads on numbers they call, reach an operator and ask a query.

But testing these agents that have dialpad tools has been notoriously hard on vapi.
The way we tested was to create another virtual number (number B) that had an IVR menu, if you pressed on a specific key on the IVR it would navigate to a test phone number we had (number C). The idea was that the voice agent should hear a human voice on number C and start speaking.

but when the number was transferred (to number C) the vapi agent was not able to hear the audio from number C, i think the media streaming on vapi had some issue with the audio coming in from number C and it is corrupted in the recordings. We tried a bunch of different virtual numbers for number B, quo, signalWire, twilio, all failing.

All to ask, is this a Vapi issue, or does every voice agent platform have issues with this. Has anyone built outbound voice agents that can actually navigate dialpads and then speak to operators after?

2

I am not sure that I am fully understanding your use-case. Is this basically the issue you are having?


sequenceDiagram
    autonumber
    participant Agent as Voice Agent<br/>(Number A)
    participant Plat as Platform media stack<br/>(SIP + RTP)
    participant B as IVR<br/>(Number B — Twilio/SignalWire/Quo)
    participant C as Test human<br/>(Number C)

    Agent->>Plat: start outbound call
    Plat->>B: INVITE (SDP offer)
    B-->>Plat: 200 OK (SDP answer: B's RTP addr, SSRC₁)
    B-->>Agent: IVR menu audio ("Press 2 for operator")
    Agent->>Plat: send DTMF "2" (RFC 4733)
    Plat->>B: DTMF events
    Note over B,C: B transfers the call leg to C<br/>(provider bridge / re-INVITE / REFER)
    B->>Plat: re-INVITE (new SDP: C's media addr) — or silent bridge
    C-->>Plat: RTP from NEW source (new IP:port, SSRC₂, maybe new codec)
    rect rgb(255, 230, 230)
        Note over Plat: ❌ Failure point:<br/>media stack still latched to SSRC₁ / old address,<br/>or ignores the re-INVITE →<br/>inbound audio dropped or decoded wrong ("corrupted")
    end
    C--xAgent: human voice never reaches agent
    Note over Agent,C: ✅ Expected: stack accepts new RTP source,<br/>applies re-INVITE SDP, agent hears human and speaks

If this is close to your use case, I don’t think there is any “inherent” problem in voice agents. I expect the issue is more related to the “flexibility” of the framework you are using.

You should be able to create a LiveKit agent and test it out pretty easily.

You may also find it helpful to explore LiveKits Automatic Machine Detection (AMD) functionality