Problem Statement
Inbound SIP calls require immediate pod availability. Unlike HTTP requests that can be queued/retried, a SIP INVITE expects a response within seconds — if no agent pod is free, the SIP gateway returns 486 Rejected and the call is lost permanently.
What We Tried (Reactive Scaling)
Attempt 1: Kafka Lag Trigger
- Trigger:
lagThresholdondev-call-requeststopic - Problem: Inbound SIP calls don’t go through Kafka — they arrive directly via LiveKit SIP gateway. Kafka lag is only relevant for outbound calls. Additionally, the consumer group had never committed offsets, causing phantom lag that scaled to 24 pods instantly.
Attempt 2: agent_active_calls Prometheus Metric
-
Trigger:
agent_active_callsper pod (threshold=1) -
Problem: This metric only increments after a call successfully connects. The sequence is:
- SIP INVITE arrives
- Pod picks up call
- Call connects
active_callsincrements- KEDA detects metric change (5-15s polling delay)
- Scales up
By step 5, the next incoming call has already been rejected because no free pod was available. The metric fires too late.
Attempt 3: livekit_participant_total
- Trigger: Total participants / 3 (each call = SIP + agent + egress)
- Problem: Same timing issue — participants increment after the call is already assigned to a pod. When the 3rd SIP caller joins (7th participant), the call is already being routed. If no pod is free, it’s rejected before the metric even changes.
Attempt 4: rate(livekit_sip_invite_requests_raw[1m])
-
Trigger: SIP invite rate per minute
-
Problem: While this fires earlier (at the SIP gateway level), KEDA still needs:
- 5s polling to detect the metric
- 15-30s for Karpenter to provision a node (if needed)
- 10-15s for pod startup + readiness
- Total: 30-60s delay
A SIP INVITE timeout is typically 3-5 seconds. Reactive scaling will always be too slow.
Why Reactive Scaling Fails for SIP
SIP INVITE lifecycle:
t=0s INVITE arrives at SIP gateway
t=0.1s Gateway looks for available agent pod
t=0.2s If no pod free → 486 Rejected (CALL LOST)
t=3s SIP timeout (caller gives up)
KEDA scaling lifecycle:
t=0s Metric changes
t=5s KEDA polls and detects change
t=5s KEDA updates HPA desired replicas
t=6s Scheduler tries to place pod
t=6-60s Node provisioning (if no capacity)
t=60-90s Pod starts, pulls image, passes readiness
⚠️ 60-90 seconds vs 0.2 seconds required
The fundamental mismatch: SIP is synchronous and immediate; Kubernetes scaling is asynchronous and slow.
Solution: Pre-warm Scaling
Instead of reacting to calls, pre-provision pods ahead of demand.
Current Configuration
minReplicas: 4 # Always 4 warm pods ready
maxReplicas: 100 # Can scale up for sustained load
How It Works
- Warm pool: Always maintain N pods ready to take calls immediately
- Scale triggers (CPU 40% + SIP invite rate): Still present to scale the warm pool UP during sustained load
- Set
minReplicas= expected peak concurrent calls for the test
Sizing the Warm Pool
minReplicas = expected_concurrent_calls + buffer
Example:
- Load test: 10 concurrent calls → minReplicas = 12
- Production estimate: 50 concurrent → minReplicas = 55-60
Trade-offs
| Approach | Pros | Cons |
|---|---|---|
| Reactive | Cost efficient (0 idle pods) | Calls rejected during scale-up window |
| Pre-warm | Zero call rejection | Higher idle cost |
For voice/SIP workloads, the cost of a rejected call (lost customer, failed SLA) far outweighs the cost of idle pods.
Reactive Scaling Still Has a Role
Reactive triggers (CPU, SIP invite rate) serve as a top-up mechanism:
- If sustained load exceeds
minReplicas, KEDA scales above the minimum - This handles gradual ramp-up (e.g., morning traffic increase)
- But the warm pool handles burst/instant demand