How do you run self-hosted STT for live calls and batch transcription - shared GPUs or separate pools?

Self Hosting
1

Hi all,

I am working on inference infra for self-hosted speech models on K8s (Whisper-class STT on GPUs) and trying to understand how people handle workloads in production.

The setup I am curious about: the same STT model serving both live sessions (agents, calls - latency-sensitive) and batch jobs (recordings, meeting files -throughput-oriented).

A few questions for anyone running this:

  1. Do you share GPU pods between live and batch traffic, or run separate pools?
  2. If separate: how utilised is each pool in practice? Does the live pool sit mostly idle off-peak?
  3. If shared: have you seen live-session latency spike when a long batch job lands on the same pod? How did you notice, and what did you do about it?
  4. How are sessions assigned to STT pod today - plain K8s Service, least-connections, something custom>

Not selling anything. I’m trying to work out whether routing on queued-audio rather than connection count is a real gap or a solved non-problem. Happy to share what I find.

2

@Abhishek_Karki, For the LiveKit part of this, live agent sessions are not assigned by connection count. The agent worker reports a load value from load_fnc, and above load_threshold the server marks that worker unavailable and stops routing new jobs to it (worker.py#L188, threshold #L192-L193). The default load is CPU, but load_fnc is any callable, so you can report GPU pressure or STT queue depth instead. That makes live-session-to-worker assignment track the signal you care about, which is the “route on queued audio, not connections” model you describe:

from livekit.agents import WorkerOptions

# return your real pressure (GPU util or STT queue depth) as 0.0..1.0
def worker_load() -> float:
    return stt_queue_depth() / MAX_QUEUE

WorkerOptions(entrypoint_fnc=entrypoint, load_fnc=worker_load, load_threshold=0.7)

Two limits. Custom load_fnc / load_threshold apply to self-hosted workers only; LiveKit Cloud reverts both to the default CPU load (#L580-L589). And this only decides which agent worker a live session lands on; it does not schedule inside your Whisper GPU pool, and batch jobs (file or recording transcription) never pass through this dispatch. So shared-vs-separate pools and batch-on-live interference sit in your own inference layer. The usual fix is to keep batch off the live GPUs (separate pool) or give live requests queue priority, because one worker processing a long file holds the GPU and head-of-line-blocks short live chunks.