Hi all, are there plans to include this plugin from this PR in next releases?
main ← szhaomsft:xinran/realtime_model
opened 11:14AM - 02 Feb 26 UTC
## Description
This PR adds support for Azure Voice Live, enabling real-time … speech-to-speech conversations through the Azure plugin.
Fix https://github.com/livekit/agents/issues/4716
### What's New
**Azure Voice Live Integration**
- New `RealtimeModel` class providing end-to-end speech-to-speech capabilities
- Full bidirectional audio streaming
- Server-side VAD (Voice Activity Detection) with configurable thresholds
- Automatic reconnection handling for connection resilience
### Features
- **Speech-to-Speech**: Direct audio input/output without separate STT/TTS pipeline
- **Function Calling**: Built-in tool use for agentic workflows
- **Multilingual Support**: Works with Azure's multilingual neural voices (e.g., `en-US-AvaMultilingualNeural`)
- **Interruption Handling**: Graceful handling of user interruptions during responses
- **Metrics Collection**: Token usage and TTFT (Time to First Token) tracking
- **Debug Mode**: Optional audio saving per turn for debugging (`save_audio_per_turn=True`)
### Usage
```python
from livekit.agents import Agent, AgentSession
from livekit.plugins import azure
session = AgentSession(
llm=azure.realtime.RealtimeModel(
voice="en-US-AvaMultilingualNeural",
)
)
await session.start(room=ctx.room, agent=Agent(instructions="You are helpful."))
```
### Environment Variables
```bash
export AZURE_VOICELIVE_ENDPOINT=https://<region>.api.cognitive.microsoft.com/
export AZURE_VOICELIVE_API_KEY=<your-speech-key>
```
### New Dependencies
- `azure-ai-voicelive[aiohttp]>=1.0.0`
- `azure-identity>=1.15.0`
---
<a href="https://livekit.devinenterprise.com/review/livekit/agents/pull/4693">
<picture>
<source media="(prefers-color-scheme: dark)" srcset="https://static.devin.ai/assets/gh-open-in-devin-review-dark.svg?v=1">
<img src="https://static.devin.ai/assets/gh-open-in-devin-review-light.svg?v=1" alt="Open with Devin">
</picture>
</a>
Looks like that PR is still open, so unless it is merged, it will not be part of the next agents release. The PR is has a lot of comments, so it’s unclear to me what the status is - looks like it is waiting for someone on eng to review