Answering Machine Detection latency concerns

Agents
1

Is anyone experiencing high latency with AMD setup. I am using gemini-3.1-flash-lite (LLM) and deepgram (STT)(enhanced-phonecall) model for detection, and I am consistently observing latency of around 1.3-1.7 seconds. In the AMD benchmarks, the P50 median latency was mentioned as ~840ms. Is anyone able to achieve this, and if yes, can you please share your setup. Also, I am using the default value for human_speech_threshold (2.5 secs), and human_silence_threshold (0.5 sec).

Adding a log for reference -

AMD Result - type=‘amd_prediction’ speech_duration=0.48259925842285156 category=<AMDCategory.HUMAN: ‘human’> reason=‘llm’ transcript=‘Hello?’ delay=1.770613670349121

2

@Harsh_Mahajan, One important things, Your log confirms you are on the native 1.5.9 implementation, but it also reveals exactly where your 1.7s latency is coming from.

Because your log shows reason='llm' and transcript='Hello?', the fast-path audio heuristic was bypassed. Per the framework design, when an STT engine successfully yields a transcript, AMD routes it to the LLM to verify the context.

The culprit is Deepgram’s enhanced-phonecall model. This is a legacy Deepgram model tier (the lineage goes Enhanced >> Nova >> Nova-2 >> Nova-3 >> Flux). The legacy Enhanced models have a high internal endpointing penalty (it waits significantly longer to ensure the speaker is done) before finalizing a transcript. Combined with the default network round-trip, you get 1.7s.

To drop down to the ~840ms P50 benchmark, you need to tighten your VAD/Turn handling and use a faster STT pipeline. Here is the optimized configuration for 1.5.9:


from livekit.agents import AMD, TurnHandlingOptions

# 1. Force LiveKit's VAD to hand off speech frames aggressively fast
turn_options = TurnHandlingOptions(
    min_endpointing_delay=0.1,
    max_endpointing_delay=0.4,
)

await session.start(
    agent=agent, 
    room=ctx.room, 
    turn_handling=turn_options
)

# 2. Execute AMD natively (Ensure your global STT plugin is upgraded)
async with AMD(session, participant_identity=participant_identity) as detector:
    await ctx.api.sip.create_sip_participant(...)
    await ctx.wait_for_participant(identity=participant_identity)
    
    result = await detector.execute()

Some of potential fixes can be:

  • Swap your session’s STT from deepgram (enhanced-phonecall) to standard deepgram (nova-3) or

  • Even better for Voice Agents, Deepgram recently launched flux-general-en, which is built explicitly for real-time voice bots and features model-integrated end-of-turn detection to minimize conversational latency.

  • Use the LiveKit Cloud default (cartesia/ink-whisper). The enhanced phone model is simply too slow for sub-second outbound detection workflows.

3

Hey Harsh, those figures, from Detect voicemail and IVR with outbound phone agents | LiveKit, were achieved using the default model and STT:

The best-performing pairing is google/gemini-3.1-flash-lite with cartesia/ink-whisper, which are our default settings for AMD when LiveKit Cloud is used: