Looking for tuning recommendations for single server self-hosted LK for SIP inbound

Self Hosting
1

Hi all — we’re building a self-hosted LiveKit SIP-based voice-agent deployment that needs to scale, and as part of that we’ve been doing some load testing and tuning. We’d really appreciate a sanity check on our setup and any suggestions for squeezing more concurrent sessions out of the hardware.

**Hardware / topology (single node):**

- AWS EC2, **8 vCPU (aarch64 / Graviton), ~15 GiB RAM**, Ubuntu.

- All LiveKit components co-located on the one box: **livekit-server (SFU), livekit-sip, and the Python agent worker**, plus a local Redis. The SIP↔SFU media path is loopback.

- The SIP/RTP load generator runs on a **separate** host, so the numbers below are the SUT’s own load.

**Software:**

- livekit-server **1.13.4**

- livekit-agents **1.6.6** (Python 3.11), livekit-plugins-deepgram, silero VAD, and the `turn_detector` EnglishModel

- livekit-sip built from source (arm64), `lk` CLI 2.18

- Worker launched with `agent.py start` (production mode)

**Workload:** inbound SIP calls → livekit-sip → SFU → a Python `AgentSession` per call (STT + LLM + TTS + silero VAD + turn detection). ~3-minute, 8-turn conversations. To keep the load test deterministic, we stub the STT/LLM/TTS vendors with **local mocks** (fixed injected latencies) so the numbers reflect the LiveKit runtime under load rather than vendor round-trip time.

**Changes we’ve already made from defaults:**

1. **`rtc.use_ice_lite: true`** — this was essential. Without it, livekit-sip attempted ICE gathering against public STUN (`stun.l.google.com`) to negotiate the loopback SIP↔SFU WebRTC transport; on this box those queries time out and every INVITE past ~14 concurrent got `486`. ice-lite (host candidates only) fixed it. (Also `rtc.use_external_ip: false`, `bind_addresses: [127.0.0.1]`, node IP auto-detected to the private address, `rtc.port_range 50000–50200`, `tcp_port 7881`.)

2. **Worker admission opened up:** `load_threshold` 0.7 → **0.95**, `num_idle_processes` 8 (=cpu_count) → **24**.

3. `job_executor_type` left at the default (**process**, fork-per-job).

4. `preemptive_generation` toggled while testing.

**What we observe:** the box tops out around **~25 concurrent sessions** on 8 vCPU, and this is essentially the same whether worker admission is at the default 0.7 or opened to 0.95 — at 0.7 it cleanly refuses (`486`) at ~70% CPU; at 0.95 it accepts more but new-call setup then times out.

**Where we’d love your input:**

1. Are there worker or runtime settings we’re missing to raise **sessions-per-core**?

2. Would **`job_executor_type = thread`** (ThreadJobExecutor) meaningfully help throughput here vs the default process executor, given the GIL and our per-call silero VAD inference — or does the process executor remain the right choice at this scale?

3. Guidance on **`num_idle_processes`** / prewarming for bursty SIP arrival — is 24 sensible on 8 cores, or counterproductive?

6. Any known agent-runtime CPU optimizations (model/config flags, disabling features we don’t need) for a mock-backed pipeline agent?

Any corrections to the above or config we should change are very welcome — we want to make sure we’re configuring this well before we scale it out. Thanks!

2

@beachdog, Your ~25 ceiling is CPU-bound and matches LiveKit’s own sizing, not a misconfig: the docs rate an agent server at 4-cores/8-GB for 10 to 25 concurrent jobs (deployment), and you are co-locating the SFU, SIP, and Redis on the same 8 vCPU, so the worker effectively has about 4 cores. That is right where 25 lands.

Directly on your questions:

  • Leave load_threshold at 0.7. Admission is CPU-gated (worker.py#L82), so 0.95 just accepts past saturation and setup times out, exactly what you saw. Stay on the process executor; thread is only the Windows default (a BrokenPipeError workaround, worker.py#L126) and runs every job in one interpreter, so per-call Silero VAD serializes on the GIL.

  • Drop num_idle_processes from 24 to about cpu_count: the prod default is CPU-count based (worker.py#L206) and idle processes are prewarmed forks that only hide burst cold-start, so 24 oversubscribe CPU and RAM for no steady-state gain.

  • On runtime CPU, your turn detector already runs in one shared inference process, not per call (turn_detector/base.py#L46); only Silero VAD runs per job (silero/vad.py#L145), so it is the single inference cost that scales with concurrency (drop to VAD-only EOU if you do not need EnglishModel).

from livekit.agents import WorkerOptions, cli

cli.run_app(WorkerOptions(
    entrypoint_fnc=entrypoint,
    load_threshold=0.7,      # CPU-gated admission; higher just degrades accepted calls
    num_idle_processes=8,    # ~cpu_count; prewarm only hides burst cold-start
    # job_executor_type defaults to process on Linux, leave it
))

The real levers are topology (agent worker on its own node, off the SFU/SIP box) and horizontal scale-out (N workers behind one server, autoscaled on CPU). Past ~25 on a shared 8-vCPU node you are fighting the CPU budget, not a config gap..

3

Thanks @Muhammad_Usman_Bashir this is much appreciated!