Using livekit.agents.llm.RealtimeModel with liteLLM

Agents
, ,
1

I’m trying to use the openai plugin to run a RealtimeModel Agent, but with a service behind a LiteLLM proxy. The .js code here would run normally, so in principle I can call openai’s gpt-realtime model. However, I cannot set up livekit.agents.llm.RealtimeModel properly to get it to work. I use:

llm = openai.realtime.RealtimeModel(
    model="gpt-realtime",
    base_url=OPENAI_BASE_URL,  # Base URL without /realtime path
    api_key=OPENAI_API_KEY,  # May not be needed with custom headers
)

where OPENAI_BASE_URL is something like ws://0.0.0.0:4000/v1. I see that it seems to require the headers:

    headers: {
        "api-key": `sk-1234`,
        "OpenAI-Beta": "realtime=v1",
    },

so I tried also:

headers = {
    "api-key": OPENAI_API_KEY,
    "OpenAI-Beta": "realtime=v1",
}

async with aiohttp.ClientSession(headers=headers) as http_session:
    print("Creating RealtimeModel with custom http_session...")

    llm = openai.realtime.RealtimeModel(
        model="gpt-realtime",
        base_url=OPENAI_BASE_URL,  # Base URL without /realtime path
        http_session=http_session,
        api_key=OPENAI_API_KEY,  # May not be needed with custom headers
    )

But I’m getting the same error.

Actually, the above runs fine. I just get a timeout error when I run something like:

session = AgentSession(llm=llm)
await session.start(Agent(instructions="You are a test assistant."))
result = await session.generate_reply(
    instructions="Greet the user and offer your assistance. You should start by speaking in English."
)

So apparently, the server has no response. But everything is quite opague, so I’m not quite sure how the headers and everything are sent to the LiteLLM gateway.

Thanks in advance for your help!

2

Really interested if others in this forum have a solution or workaround, but I fed your issue into Devin and it gave a plausible reason for what you’re seeing: Search | DeepWiki

3

Thanks for the help, but I’m not sure Devin made the right diagnosis, cos LiteLLM also supports using "Authorization”: “Bearer sk-1234”.