Per-session CPU burst from a plugin's native model being built lazily on the first audio frame (AI_Cousitcs) — any pattern for prewarmable per-session init?

Agents
, ,
1

Description:
A noise-cancellation plugin we use builds its native model lazily on each session’s first audio frame rather than ahead of time, causing a short but real CPU burst at the start of every session. Looking for whether there’s an existing pattern/hook in the agents framework (or plugin ecosystem generally) for doing this kind of per-session-but-heavy initialization earlier — ideally during prewarm — instead of on the first real frame.

Problem text:

We run livekit-agents (Python) as a voice agent using ai-coustics for noise cancellation (ai_coustics.audio_enhancement(model=EnhancerModel.QUAIL_VF_S)). We noticed a consistent, short CPU spike at the start of every new session and traced it to the plugin’s lazy initialization.

Looking at AICousticsAudioEnhancer._process():

if self._enhancer is None or (
    # implicitly recreate on sample rate/channel change
    ...
):
    self._settings = EnhancerSettings(
        sample_rate=frame.sample_rate,
        num_channels=frame.num_channels,
        samples_per_channel=frame.samples_per_channel,
        model=self._model,
        ...
    )
    self._enhancer = Enhancer(auth_mode, self._settings)   # native model construction happens here

The actual native model (a compiled Rust core, constructed via FFI) is only built on the first real audio frame of a session — not when the AICousticsAudioEnhancer/AudioInputOptions object is constructed, and not at room-join time. In our setup this lines up exactly with a CPU spike each time a new session starts.

We prewarm other per-process resources ahead of time via WorkerOptions.prewarm_fnc (e.g. warming our own service clients), but we can’t do the same for this model, because its settings (sample_rate, num_channels, samples_per_channel) are derived from the actual incoming audio frame — they aren’t known until a real session’s real track is live. So today the only two options we can see are: (a) eat the cost on the first frame, as it does now, or (b) build some kind of throwaway “warm-up” instance during prewarm, which only helps if there’s shared/cacheable state at the process level (model weights loaded into page cache, a shared native runtime, etc.) rather than purely per-instance cost — something we have no visibility into since the plugin is a closed-source compiled component.

We did notice the plugin exposes update_stream_info() and update_credentials() methods for re-pointing an already-constructed Enhancer at a different stream/credentials, which suggests the native core might be designed to support reuse across streams rather than one-construction-per-session — but we haven’t found guidance on whether that’s an intended/supported usage pattern, or whether it’s safe to do from application code (thread-safety, state leakage between sessions, etc.).

Questions for the community:

  • Is there an existing pattern or hook — in livekit-agents itself, or in how plugin authors structure things — for per-session initialization that still needs to run ahead of time (e.g., during prewarm) rather than lazily on the first frame?
  • Has anyone tried pooling/reusing a plugin’s native processing instance across sessions via re-pointing APIs like update_stream_info()/update_credentials(), instead of constructing fresh per session? Any gotchas?
  • More generally for plugin authors doing native/FFI-backed per-session setup: is “lazy construction on first frame” the recommended approach, or is this specific to how this plugin happens to be built?

Environment: livekit-agents (Python), one ai-coustics enhancer constructed per session via AudioInputOptions(noise_cancellation=...).

2

The part you can’t see into the plugin for, you can settle from the framework side — process state is shared across sessions, so a throwaway warm-up is worth testing.

prewarm_fnc runs inside _JobProc.initialize(), which creates the JobProcess once and then hands control to entrypoint(). That entrypoint loops on StartJobRequest messages, so one process serves many jobs sequentially and the same JobProcess — and its userdata — persists across all of them. Prewarm is per-process, not per-session.

So the question isn’t whether warming can be shared. It’s whether the cost you’re paying is process-level (loading the native library and model weights into the process) or per-instance (allocating a new enhancer around already-loaded weights). You can find that out without any visibility into the plugin:

Build two enhancers back to back inside prewarm_fnc and time both. If the second is much faster than the first, the expensive part is process-level and a throwaway warm-up during prewarm will kill your per-session spike. If both cost the same, it’s per-instance and warming buys you nothing — at which point pooling via update_stream_info() is the only route left.

Five minutes, and it tells you which of your two options is actually available.

On whether lazy-on-first-frame is the recommended FFI pattern — I’d say the framework’s shape argues against it. prewarm_fnc plus JobProcess.userdata exists precisely so heavy init lands at process start rather than in the audio path, and initialize() even imports a _preload module described in the source as the framework’s own warm-up. Deriving settings from the first frame is a real constraint, but it’s a plugin design choice rather than a platform one.