Video Frame Metadata support for Golang SDK

Server SDKs
1

Hi LiveKit community!

I’m interested in the recently introduced video frame metadata feature, and I’m curious about the support for it in Golang SDK, is it on the roadmap?

2

Hi @kangjingyang , I don’t believe there are any current plans but I have reached out internally to hopefully get you an authoritative answer.

3

Thanks Darryn, would be great to have this. I can help with early alpha/beta testing or even with the development.

Also are there any reason for the 232 bytes limit? Are there any technical reason or just a product feature choice?

4

Hi @kangjingyang can you describe how you would want to use the Go SDK to publish video w/ frame metadata? We do have support for reading/writing the metadata in the SDK, however since there isn’t a direct way to publish video tracks from frames, it’s there as a way to forward frame metadata from existing H264/H265 streams.

The 232 byte limit is because in the trailer protocol, we used a uint8 as the length field for the total trailer, which allows for 232 bytes of user data after the necessary framing. We don’t intend for this to carry bulky data attached to video frames, which is why we decide for the length limit.

5

Hi @David_Chen the reason why I’m interested in this feature is because we are building livekit integration of Spatius, which is a real time lip-sync avatar service. What’s different from existing avatar services is that we don’t stream video frames, instead we stream motion data and renders the avatar client side. So for the integration, we need a good way to transport non-video motion data through livekit, and also make sure it’s synced packet level with audio so the movement of avatar doesn’t drift from the audio playback.

Currently our method is to build a custom payload mechanism in VP8 video track. We pack motion data in RTP payload (server side, using the Golang SDK) to “trick” browser depacketizer to think it’s encoded VP8, and in client side use encoded transform (same browser API used internally by livekit web client SDK to implement video frame metadata) to unmarshall it to motion data and send to our core rendering SDK. https://github.com/spatius-ai/avatarkit-web-rtc/tree/main/src/providers/livekit is the implementation of client side unmarshall & depacketizing logic.

Though it works, it’s not perfect since it’s quite a hacky way to achieve this. So I’m very interested to explore what the standard video frame metadata feature can do. Currently there seems like two blockers for us

  1. Golang SDK support (seems like not a strong blocker though, since we already compose each RTP packet ourself anyway)
  2. The 232 bytes size limit for each frame. Currently our motion data is in 25 FPS and takes 480 bytes per frame. So I don’t know if it’s possible for us to achieve this by negociating a video track in higher FPS, for example 75+ FPS.

There are obvious benefits to use standard video frame metadata feature in LiveKit in the long run, for example we won’t need to implement custom depacketizing logic for each platform’s client SDK (if video frame metadata will be supported by livekit on each platform). Currently we only implemented web SDK, and find it a much hevyer work to implemnt this for iOS/Android/Flutter/React Native. That’s why I’m realy interested and love to do whatever I can do to test & play with it as early as possible.