how‐to‐build‐cartwheel‐ffmpeg‐libtorch‐on‐linux

How to build ffmpeg-libtorch

Install ipex

1. install oneAPI base toolkit: https://www.intel.com/content/www/us/en/developer/tools/oneapi/base-toolkit-download.html
2. install ipex follow official guide to install ipex. https://github.com/intel/intel-extension-for-pytorch
   (for example: python -m pip install torch==2.0.1a0 torchvision==0.15.2a0 intel_extension_for_pytorch==2.0.110+xpu -f https://developer.intel.com/ipex-whl-stable-xpu)

setup oneAPI environment

$ source /opt/intel/oneapi/setvars.sh

Build ffmpeg-libtorch

$ git clone https://github.com/intel/cartwheel-ffmpeg --recursive
$ cd cartwheel-ffmpeg
$ git submodule update --init --recursive
$ cd ffmpeg
$ git am ../patches/*.patch

Build. (assume pytorch and ipex is installed under "/usr/local/lib/python3.10/dist-packages")

$ ./configure --enable-libtorch --extra-cflags='-I/usr/local/lib/python3.10/dist-packages/torch/include/torch/csrc/api/include -I/usr/local/lib/python3.10/dist-packages/torch/include' --extra-ldflags='-L/usr/local/lib/python3.10/dist-packages/torch/lib -L/usr/local/lib/python3.10/dist-packages/intel_extension_for_pytorch/lib'
$ make && make install

Convert pytorch model to torchscript model

Convert to torchscript

ffmpeg example command

$ ffmpeg -i input.mp4 -vf format=rgb24,dnn_processing=dnn_backend=torch:model=model.pt:input_nb=9:backend_configs=device=xpu,format=yuv420p output.mp4