Fined tuned open source model and use its textual generation to generate audio with kokoro. Now just wanna link it to real time avatar. But ones that I found is either high latency or dont work so well. Come to find LiveKit on youtube. It seem to offer audio->real time avatar. But the site is like a maze, my head is spinning right now. Used to work with Huggingface models and fine tuning. Never really used this kind of stuff. HELP!
Hi, welcome to LiveKit.
The best way to get started with an Avatar & LiveKit would be as follows:
- Follow our voice AI agent quickstart guide: Voice AI quickstart | LiveKit Documentation . That will get you to a point where you can speak with a voice agent.
- Follow our react quickstart to get a front end up and running, React voice AI quickstart | LiveKit Documentation , you will need this to render the Avatar. We support a LOT of client front-ends, but this is the easiest way to get started.
- Update your agent code to enable the Avatar. The first bullet above pulled down our agent starter, so it should be a case of uncommenting the code at https://github.com/livekit-examples/agent-starter-python/blob/main/src/agent.py#L138 and substituting with the avatar provider and key of your choice. Supported providers are here: Virtual avatar models overview | LiveKit Documentation
More docs, if you didn’t see it already: Virtual avatars | LiveKit Documentation
Darryn’s steps are the right path, and for your exact stack there is an official example that wires Kokoro: examples/other/kokoro_tts.py. It runs Kokoro through its OpenAI-compatible server (Kokoro-FastAPI) via the openai TTS plugin:
from livekit.plugins import openai
tts = openai.TTS(
model="kokoro",
voice="af_alloy",
api_key="not-needed",
base_url="http://localhost:8880/v1", # your Kokoro-FastAPI server
response_format="wav",
)
Your fine-tuned LLM plugs in the same way: serve it behind an OpenAI-compatible server (vLLM, Ollama) and point the openai LLM plugin at it with base_url (openai llm.py), for example
openai.LLM(base_url="http://your-model-server/v1", model="your-model"). So the whole thing is your LLM plus Kokoro in an AgentSession, then enable the avatar exactly as Darryn described. There is no native Kokoro plugin yet, only an open request for one (agents#6184), but the example above is the working path today.