Hey everyone!
I wanted to use smart-turn model for my language in a LiveKit agents. Found out there wasn’t any straightforward way to plug a Smart Turn model directly into LiveKit, so I ended up building it.
I’m publishing it as standalone pypi plugin:
It handles the audio preprocessing, ONNX inference, resampling, model loading, and LiveKit turn-detector integration, so you can use an upstream Smart Turn model, a fine-tune, or your own ONNX checkpoint directly with AgentSession.
Nice work. Two things that might help with positioning and adoption.
The audio turn detector covers 14 languages: English, Arabic, German, Spanish, French, Hindi, Indonesian, Italian, Japanese, Korean, Dutch, Portuguese, Turkish and Chinese. If yours is outside that list, lead with it, since that’s a gap nothing else fills.
There’s a second angle worth claiming. The docs say the text detector is kept around only for cases where you can’t use LiveKit Inference and need a fully open-weights, self-contained option, and that it’s being removed in 2.0. That leaves a real hole for anyone air-gapped or on a strict data-residency requirement. If your plugin is fully local ONNX with no callout, say so explicitly, because that audience will be looking for a replacement.
For benchmarks, LiveKit publishes eot-bench, an open harness that simulates live turn-taking decisions under production endpointing policies. Numbers from their own harness will carry more weight than a custom setup.
Two implementation details worth matching. unlikely_threshold accepts either a scalar or a dict keyed by language code, so supporting both keeps you drop-in compatible. And the audio detector requires VAD with min_silence_duration of at least 0.25s, raising a ValueError below that. Worth checking whether your plugin has a similar floor and failing loudly if so, rather than degrading quietly.
Thanks, this is really helpful.
Yes - my language is outside of LiveKit’s 14. I’ll lead with that.
On offline: it’s fully local ONNX, no callout at inference. Weights download from HuggingFace once and cache; local_only=True, HF_HUB_OFFLINE=1, or an explicit model_path= cover air-gapped and data-residency setups. I’ll make that explicit in the README.
On eot-bench, it’s a plugin - doesn’t bundle any weights; it loads whatever Smart Turn checkpoint you point it at, downloaded at runtime.
The min_silence floor is already covered: LiveKit raises a ValueError below 0.25s for any streaming detector, so it fails loudly. It’s documented.
Good catch on unlikely_threshold - I match the protocol method, but only take a scalar at construction. Will adding dict support.
To be precise on eot-bench: it’s the evaluation harness rather than anything you’d bundle. It benchmarks end-of-turn models under production endpointing policies, simulating the turn-taking decisions a real agent makes, so you’d be running your plugin through it and getting numbers comparable to LiveKit’s own.
One thing to check before investing time there. LiveKit also publishes eot-evals, their English and multilingual end-of-turn test datasets. Worth confirming whether your language is in the multilingual set, because if it isn’t you’d have the harness but nothing to run it against. The Smart Turn side may have something usable too, and you’ll know that ecosystem better than I do.
If it turns out there’s no suitable eval data and you end up labelling a set yourself, publishing it alongside the plugin would be worth as much as the code. It lets people verify your numbers and fine-tune their own checkpoints, and eval data for underserved languages tends to be harder to come by than model code.
The language is Tamil - which is neither among LiveKit’s 14 languages nor Smart Turn’s 23 languages, so ended up using fine-tuned model for Tamil. Checked eot-bench-data as well and it’s the same 14 as expected.
I do already have a public dataset, but it’s the different shape: ~18k pre-cut 8-second clips with binary labels, where eot-bench wants complete user turns annotated with every silence span.
The source audio is real Tamil telephone conversation (CC BY 4.0), I still have the full calls the clips were cut from, so I’d be happy to contribute a Tamil dataset in the eot-bench format if you’re accepting any contributions.
Hi @Santhosh , this sounds familiar. I don’t think it was with yourself, but I remember a previous conversation where a developer had created a custom LiveKit turn detector for a language we didn’t support. Frustratingly, I can’t find that conversation and it might have been in the Slack community where conversations aren’t stored forever.
Thanks for contributing.
You might want to modify the title of this topic to include that info - we get a lot of organic traffic from Google searches, so it would help folks find your plugin who are looking for Tamil.
