[feature] Add Stream API for capturing video frames from WebRTC MediaStreamTrack

[YouheiShikano]

Is your feature request related to a problem? Please describe.
Currently, when trying to analyze video frames from WebRTC MediaStreamTrack in real-time (e.g., for ML processing), we need to use Timer to periodically capture frames using captureFrame(). This approach has several issues:

• High CPU/Memory usage
• UI stuttering
• Potential frame drops
• Inconsistent frame rate
  Describe the solution you'd like
  Add a new Stream-based API to capture video frames from MediaStreamTrack that:

1. Provides frames as a Stream at the native frame rate
2. Integrates directly with the native WebRTC frame callback system
3. Has an API similar to:

Stream<VideoFrame> getVideoFrameStream() async {
// Implementation that hooks into native frame callbacks
}

This would allow more efficient frame processing without Timer polling.
Describe alternatives you've considered

1. Current Timer-based approach with captureFrame()
2. Custom native implementation per platform
3. Using platform channels with event streams

The Stream API would be the most elegant and performant solution.

Additional context
Looking at the current implementation in the codebase:

1. The current implementation uses Timer-based approach:

  captureImageTimer = Timer.periodic(
   const Duration(milliseconds: 200),
  () async {
    if (!mounted) return;
    await caputureImage(videoTrack);
   },
);

  Future<Image> captureImage({
    required VideoTrack videoTrack,
  }) async {
    try {
      final mediaStreamTrack = videoTrack.mediaStreamTrack;

      final imageByteBuffer = await mediaStreamTrack.captureFrame();

      final imageBytes = imageByteBuffer.asUint8List();

      final image = await compute(
        decodeImage,
        imageBytes,
      );
      return image;
  }

2. The proposed Stream API could be integrated with the existing RemoteRehabilitationRoomState class, replacing the current Timer implementation:

Stream<VideoFrame> getVideoFrameStream() async {
// Implementation using native frame callbacks
}

3. Benefits of this approach:

• More efficient resource usage
• Better frame timing accuracy
• Native integration with WebRTC frame callbacks
• Reduced CPU/Memory overhead
• Smoother integration with ML processing pipeline

/cc @cloudwebrtc

[YouheiShikano]

Is this possible to implement? I'd like to mention @cloudwebrtc for their expertise.

I tried to extend the package myself by downloading it, but the dependencies with other packages seem quite strong and make it challenging to modify independently.

I'm willing to put in the effort to implement this myself, but I would greatly appreciate any advice or guidance on:

1. The feasibility of this feature
2. The best approach to implementation
3. Potential pitfalls to be aware of

Thank you for considering this request!

[cloudwebrtc]

hey @YouheiShikano , I am trying to improve this PR #657 to add a frame sink API for local or remote track, and I am also trying to write a filter example plugin so that you can follow this example to write a sink or processor for the track in native. Like ML processing

https://github.com/livekit-examples/flutter-filter-plugin-example

[YouheiShikano]

Thank you so much @cloudwebrtc for your detailed response and the great solution!

The Track Processor API looks exactly like what I needed. The implementation with native frame callbacks instead of Timer-based approach will definitely solve the performance issues I was experiencing.

I really appreciate you sharing the flutter-filter-plugin-example as well. This will be incredibly helpful as a reference for implementing custom processors for ML processing.

I'll try implementing this using your example as a guide. Thanks again for your help and for working on this improvement!

Looking forward to trying out PR #657 when it's ready!