Could we specify the codec used for decoding?

[hmaarrfk]

PyAV/av/container/input.pyx

Line 73 in ee5c40e

codec = lib.avcodec_find_decoder(stream.codecpar.codec_id)

It would be great if we could have a way to override this default.

For example, on my machine, I have:

$ /usr/bin/ffmpeg -hide_banner -decoders | grep h264
 VFS..D h264                 H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
 V..... h264_v4l2m2m         V4L2 mem2mem H.264 decoder wrapper (codec h264)
 V....D h264_qsv             H264 video (Intel Quick Sync Video acceleration) (codec h264)
 V..... h264_cuvid           Nvidia CUVID H264 decoder (codec h264)

 V..... libdav1d             dav1d AV1 decoder by VideoLAN (codec av1)
 V....D libaom-av1           libaom AV1 (codec av1)
 V....D av1                  Alliance for Open Media AV1
 V..... av1_cuvid            Nvidia CUVID AV1 decoder (codec av1)
 V....D av1_qsv              AV1 video (Intel Quick Sync Video acceleration) (codec av1)

and it would be awesome to be able to change the one that is used at runtime for performance optimization based on the user's hardware.

For example there seems to be the find_decoder_by_name function
https://ffmpeg.org/doxygen/7.0/group__lavc__core.html#gad4e08a758f3560006145db074d16cb47

I'm not sure if you have a good idea for how to do this, but it would be great if it could be done! Thanks!

[hmaarrfk]

PS. I know that you yourself are not using /usr/bin/ffmpeg in the package on pypi but I am integrating this with conda-forge's ffmpeg, where we do have some control over the number of shipped codecs. Most specifically, I am interested in playing with the intel QSV decoder! Thanks!

[jonathanhoskin]

@hmaarrfk did you ever figure this out? I am trying to solve the same problem.

[hmaarrfk]

the question for me is how to specify a good API for this.

I think a good start would be get to a point wherey ou can compile PyAV yourself, and suggest a patch.

[jonathanhoskin]

Oh ok. That's not my problem then. I am just trying to get PyAV to use NVIDIA CUDA hardware decoding but can't find any recent examples of how to do it.

[hmaarrfk]

Oh ok. That's not my problem then. I am just trying to get PyAV to use NVIDIA CUDA hardware decoding but can't find any recent examples of how to do it.

yes, this is what i was trying to do. you need to likely ask ffmpeg to do.

I don't think it is being done, you can likely force the h264_nvenc in the line I pointed to.

The question from a library perspective, is how do you do this nicely?

MP4 is a container, you don't know what the codec is until you open it. you don't know if your hardware decoder can handle it until you read the metadata (think super tiny image).

[hmaarrfk]

mind you, your ffmpeg must have been compiled with nvenc support, not sure how you are installing PyAV, but not all ffmpeg's are built equal ;)

[jonathanhoskin]

mind you, your ffmpeg must have been compiled with nvenc support, not sure how you are installing PyAV, but not all ffmpeg's are built equal ;)

Oh - I'm definitely working with a special ffmpeg build that supports cuda. If I pass the same options to PyAV it ignores them.

This CLI works:

ffmpeg -hwaccel cuvid -c:v h264_cuvid -an -rtsp_transport tcp -i rtsp://[host]:554/stream1 -f null -

This Python doesn't:

source = av.open('rtsp://[host]:554/stream1', options={'rtsp_transport': 'tcp', 'hwaccel': 'cuvid', 'video_codec': 'h264_cuvid'}, metadata_encoding='utf-8')

Am I doing it wrong?

[hmaarrfk]

Oh - I'm definitely working with a special ffmpeg build that supports cuda. If I pass the same options to PyAV it ignores them.

this is probably best for a discussion. start one. and ping me there.

please discuss how you installed both ffmpeg command line and pyav