After-hours AI agent with warm transfer while keeping existing number

Getting Started
, ,
1

I’m trying to figure out the easiest setup for businesses that want to keep their existing phone number while having an AI voice agent answer after-hours calls and warm transfer to a human when needed. The main goal is to keep the setup as low-friction as possible for the business.

The naive solution I have in mind right now is having 3 phone numbers:

  1. Original business number - remains as a client-facing interface - simple call forwarding to a middleware number I manage (2).

  2. Middleware number (likely via Twilio) - decides whether to connect with LiveKit agent or forward to a human number (3) based on business hours (probably custom business logic in a webhook).

  3. Human number - provide the business with an extra phone number for a human to pick up.

Warm transfer will transfer to the human number (3) as well.

a. Is there a better way to go about it?

b. Will the latency be reasonable? For instance, reaching a human during business hours will go through 2 additional hoops (original → middleware → human)

2

We built exactly this — here’s a simpler approach than 3 numbers:

Your 3-number architecture adds unnecessary latency and complexity. We run a production warm transfer system (GoFonIA) where the flow is:

Business number → SIP Provider → LiveKit SIP Bridge → AI Agent (single room)
↓ when transfer needed
Music Bot (hold)
↓
Outbound SIP call to human
↓
AI briefs human, human accepts
↓
Caller ↔️ Human connected

Key differences from your approach:

  1. No middleware number. LiveKit IS the middleware. The business number forwards via SIP trunk directly to LiveKit. One number, one hop.

  2. Single room architecture. All participants (caller, AI agent, hold music, human) stay in ONE LiveKit room. We control who hears whom via a 6-phase subscription matrix — no call transfers between rooms.

  3. Warm transfer flow:

    • Caller talks to AI agent
    • AI says “Einen Moment, ich verbinde Sie mit [Name]”
    • Music bot starts playing hold music (WebRTC participant)
    • AI dials human via outbound SIP
    • AI briefs human: “You have a caller on the line, they want X”
    • Human accepts → subscriptions switch → caller hears human, AI goes silent
    • Human declines → back to AI, caller never knows

  4. Latency: Business → LiveKit is one SIP hop. The “extra hoop” you’re worried about doesn’t exist because there’s no middleware number — LiveKit handles routing internally.

Tech stack:

  • LiveKit Server (self-hosted, Docker)
  • Python Agent SDK (livekit-agents)
  • Custom transfer controller (aiohttp API on port 9100)
  • Music bot (WebRTC participant publishing hold audio)
  • PM2 for process management

To answer your questions:

a) Yes, there’s a better way — skip the middleware number entirely. Let LiveKit be the decision point.

b) With your 3-number setup, you’d have ~200-400ms extra latency per hop (SIP signaling + RTP setup). With our approach it’s one SIP inbound + one SIP outbound, both happening in parallel while the caller hears hold music.

The whole thing runs on a single €5/month VPS with 3 PM2 processes. Happy to share architecture details if helpful.

3

Thanks for the detailed response.

It’d make sense to drop the middleware number if we were only talking about warm transfers, but in your approach, where would the business hours routing logic sit?