It would be ideal if we could embed json or protofuf directly into the RTP feed, thus taking advantage of existing FEC and QoS mechanisms in forwarding, however the Livekit SFU filters on WebRTC-compatible payload types.
Could the SFU perform that filtering on the outbound to a WebRTC client?
Then the Livekit SDK, and/or maybe ingress/egress, could receive and forward all payload types.
@sbarber, For arbitrary JSON/protobuf today the supported path is data packets, and the lossy mode is the low-latency option: each packet sent once, no ordering, which fits realtime telemetry [ docs.livekit.io/home/client/data/messages ]:
# lossy: sent once, no retransmit, low latency
await room.local_participant.publish_data(proto.SerializeToString(), reliable=False)
(JS equivalent: room.localParticipant.publishData(bytes, { reliable: false }).) The catch for your case: data packets ride a separate SCTP data channel, not the media RTP path, so they don’t inherit the FEC/QoS you’re trying to reach. Lossy is also capped around 1300 bytes to stay under MTU (reliable goes to 15KiB).
Forwarding in the SFU is bound to the codec set it registers with the media engine;
Opus / RED / PCMU / PCMA,VP8 / VP9 / H264 / AV1,So a payload type outside that set isn’t negotiated on either the publish or subscribe leg, not just filtered on egress [ livekit/livekit pkg/rtc/mediaengine.go ]. Arbitrary data is meant to ride the SCTP data channel instead (what publishData and the server-side data-track path use), which has its own reliable/unreliable modes but not the media RED/FlexFEC/RTX machinery [RFC 8831; RFC 2198]. Decoupling media forwarding from negotiated payload types so the SFU relays custom RTP is a real design change, and I didn’t find an open issue for it.
on Data Tracks.
Some other useful links:
Docs: Data tracks | LiveKit Documentation
Blog: Introducing data tracks: realtime streaming beyond audio and video | LiveKit
YouTube demo:
