Select video interface

[sfmig]

• @niksirbi has used sleap_io:
  • it has an excellent interface for lazy-loading videos into Python, which makes them available for indexing with numpy-style syntax.
  • It support multiple video backends ( supports reading through FFMPEG (the default), pyav, or OpenCV) and it works very well. It was a great help for this animation (each animation iteration just loads the corresponding video frame in memory, without having to load the entire video, or to extract the frames as images beforehands).

Other options:

• pyvideoreader by Jan Clemens, used in his napari-video plugin.
  • from a quick look it seems like a lightweight/simpler alternative to sleap_io, being "a numpy-like interface, using opencv as a backend".
  • Opencv has issues with seeking, the napari-video devs are open to adding or replacing the opencv backend with pyav (see discussion here)

[niksirbi]

There's also FastVideoReader - a class within the napari-pyav plugin that is mentioned in the discussion linked above. It uses pyav as a backend, and is reportedly, well, fast.

That said, for ethology I think sleap_io would make the most sense, because of the multiple backend support, and similar use-case to SLEAP (i.e. need to extract frames for labelling etc.). The other two solutions are more designed for video playback in napari.

[sfmig]

torchcodec allows for videos to be decoded as PyTorch tensors, on CPU and CUDA GPU. It has nice sampling functionalities. It's in early development stages.

It's an evolution from torchvision's read_video, which is demoed in a nice tutorial applying RAFT to predict optic flow in a video.

[talmo]

For pushing on throughput performance (to the extent that that's relevant here), I really like @sfmig's suggestion of torchcodec since they went through and did the gnarly C++ work to compile against libav/ffmpeg.

Would love to hear any experiences re: support of non-GPL formats. It looks like they dynamically link them in, but I've had mixed experiences with other libs that try to do this.

For completeness, the classic for DL-oriented workloads has been to use decord (example).

For both, I suppose the cross platform compatibility is always challenging 😦.

Also relevant: https://github.com/danionella/daio