Add ci-openrv for VFX 2024

[cedrik-fuoco-adsk]

This PR add the container image to build VFX2024 OpenRV.

OpenRV can not be built for VFX2024 right now. The work to support VFX 2024 is still ongoing (Qt6+Pyside6).
Once we are able to start a build with Qt6, I expect that there might be changes needed to Qt6 package or OpenRV container image.

Note: In theory (not tested), this container image could be use for VFX 2023 if Qt5 is installed manually (e.g. within a github action workflow) because OpenRV download and install all of its dependencies for now (expect Qt).

[cedrik-fuoco-adsk]

I wasn't really sure about that part since OpenRV is not a library like the other one.

[jfpanisset]

In versions.yaml The vfx1 / vfx2 sections of package and the vfx1 / vfx2 / vfx3 sections of versions.yaml is more about trying (and somewhat failing) to capture dependencies rather than "logical groupings": you have to build every package in vfx1 before vfx2 and you have to build images in vfx before vfx2 and vfx2 before vfx3.

Since OpenRV doesn't depend on osl or openvdb you should probably put it as the last entry in the image: vfx2 section, if that doesn't work it could go as the first entry in vfx3.

Once you've worked out everything you need for a successful OpenRV build, you will want to add openrv to the package: vfx2 section which will build a ci-package-openrv container with an OpenRV build (the ci-openrv container just has the dependencies for such a build), and add openrv to ci-vfxall/image.yaml to pull your OpenRV build into the all-emcompassing ci-vfxall image.

[jfpanisset]

In order to address:

• CVE-2024-45492
• CVE-2024-45491
• CVE-2024-45490

do you think it would be OK to change ASWF_EXPAT_VERSION from 2.5.0 to 2.6.3 and add that version to packages/conan/recipes/expat/conandata.yml?

Happy to submit that as a separate PR as well.

[cedrik-fuoco-adsk]

Changed expat to 2.6.3 and small improvements for build_openrv.sh

[jfpanisset]

Looks like the CI build is failing on base2 / Qt: there's unfortunately a chicken and egg problem where we need a release of the right version of a Conan package (here expat 2.6.3) in order for a CI build to be able to pull that Conan package and complete. That obviously needs to be fixed since it makes it impossible to test a Conan package bump from CI.

I'll submit a PR with just EXPAT 2.6.3 and make a release of that Conan package so you can pick it up for your CI runs.

[jfpanisset]

[#208] has expat 2.6.3 change, and a bunch of Python tooling updates, so you'll likely need to update your branch, sorry.

Once 208 goes in, I'll build an expat 2.6.3 Conan package and you should be able to update and build your PR. And once that's in, I'll likely make a 2024.2 release.

[cedrik-fuoco-adsk]

The checks is failing on VFX1/VFX2 pacakges with this errors:
ERROR: failed to solve: failed to resolve source metadata for docker.io/aswflocaltesting/ci-common:4-clang16: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Can we do something about that? @jfpanisset

[jfpanisset]

The aswfdocker tool has logic to determine the context it is running in and in the case of a "foreign" branch it defaults to a dummy aswflocaltesting context which is unable to access Docker images or Conan packages.

I want to fix that so that CI in "outside the repo / org" branches can work (well, modulo not having access to larger runners, which is another topic), but I need to think carefully about the implications.

For my experience, if you can do a local build from your branch, then the CI build has a good chance of working (and your local build is likely to be a lot faster anyway!).

So if you want to address what makes sense of the comments I made previously, check that a local build works, update the branch and let me know, I can merge as is, we'll let the CI builds run, and fix as needed.

This is what I use for a local build:

aswfdocker --verbose build --ci-image-type PACKAGE --group common --version 4 --target ninja --use-conan --progress plain
aswfdocker --verbose build --ci-image-type PACKAGE --group common  --target clang --version 4-clang16.1 --version 4-clang17.1 --use-conan --progress plain
aswfdocker --verbose build --ci-image-type IMAGE --group common --version 4-clang16.1 --version 4-clang17.1 --progress plain
aswfdocker --verbose build --ci-image-type PACKAGE --group base1 --version 2024 --use-conan --target blosc --target boost --target cmake --target cppunit --target expat --target glew --target glfw --target gtest --target log4cplus --target python --target tbb --target pybind11 --progress plain
aswfdocker --verbose build --ci-image-type PACKAGE --group base2 --version 2024 --progress plain --build-missing --use-conan
aswfdocker --verbose build --ci-image-type PACKAGE --group base3 --version 2024 --progress plain --build-missing --use-conan
aswfdocker --verbose build --ci-image-type IMAGE --group base --version 2024 --progress plain
aswfdocker --verbose build --ci-image-type PACKAGE --group vfx1 --version 2024 --progress plain  --target imath --target openexr --target alembic --target ptex --use-conan
aswfdocker --verbose build --ci-image-type PACKAGE --group vfx1 --version 2024 --progress plain
aswfdocker --verbose build --ci-image-type PACKAGE --group vfx2 --version 2024 --progress plain --target opensubdiv --target partio --target materialx --use-conan
aswfdocker --verbose build --ci-image-type PACKAGE --group vfx2 --version 2024 --progress plain
aswfdocker --verbose build --ci-image-type IMAGE --group vfx1 --version 2024 --progress plain
aswfdocker --verbose build --ci-image-type IMAGE --group vfx2 --version 2024 --progress plain
aswfdocker --verbose build --ci-image-type IMAGE --group vfx3 --version 2024-clang16.1 --version 2024-clang17.1 --progress plain

[cedrik-fuoco-adsk]

I updated the branch with the latest main and I was able to execute all the commands you specified without any errors.

[jfpanisset]

Let's merge as is so you can keep making progress, the comments in this PR can be addressed in a follow on

[cedrik-fuoco-adsk]

Let's merge as is so you can keep making progress, the comments in this PR can be addressed in a follow on

Which comments are you referring to? I thought that the changes were made in #208 ? @jfpanisset

[jfpanisset]

#208 addressed EXPAT 2.6.3, but not the rest of the inline comments from my initial review (see above).

For the comments about build_openrv.sh : that script would only get run in the context of building the "kitchen sink" ci-vfxall images, and thus should refer to an explicit open rv version defined in versions.yaml. The ci-openrv container that will be used by OpenRV's CI process doesn't include OpenRV itself, and it's up to the OpenRV CI to decide what to build, typically top of tree or the current branch being tested.

I'm happy to submit a PR and have you review it if you want.

[cedrik-fuoco-adsk]

I can work on a fix for the build_openrv.sh. I am not sure why, but I don't see any inline comments that you are referring to.

Were the changes addressed in the last release?

[jfpanisset]

These are the comments I was mentioning, forgot to click on Submit Review

[jfpanisset]

In scripts/base/install_yumpackages.sh we already have:

yum install ...
    mesa-libEGL-devel \
    mesa-libGL-devel \
    mesa-libGLU-devel \
    mesa-libGLw-devel \

so might as well add the additional mesa- dependencies you need there, especially if at some point we want to turn the OpenRV build into a Conan recipe.
And given that Meson and Nasm are small and bring minimal additional dependencies, let's put those in install_yumpackages.sh as well.
Since build tools like CMake and Ninja are already versioned Conan recipes, partially to better control the version we use, and partially to allow using newer versions than what's in the base OS, perhaps Meson should be as well? But that can be in a future PR.

[jfpanisset]

Not sure I understand this comment. We have a Conan PySide2 package in aswf-docker, is OpenRV using that, or does it try to pull down its own?

[cedrik-fuoco-adsk]

Currently, OpenRV does not use find_package except for like Qt and OpenGL. OpenRV build its own dependencies which includes PySide2. I encountered the same issue that build_pyside.sh script is having when Qt files are located in system files.

So I had to copy Qt outside of system files temporarily. The goal would be to allow the use of PySide from the conan package.

[jfpanisset]

In versions.yaml The vfx1 / vfx2 sections of package and the vfx1 / vfx2 / vfx3 sections of versions.yaml is more about trying (and somewhat failing) to capture dependencies rather than "logical groupings": you have to build every package in vfx1 before vfx2 and you have to build images in vfx before vfx2 and vfx2 before vfx3.

Since OpenRV doesn't depend on osl or openvdb you should probably put it as the last entry in the image: vfx2 section, if that doesn't work it could go as the first entry in vfx3.

Once you've worked out everything you need for a successful OpenRV build, you will want to add openrv to the package: vfx2 section which will build a ci-package-openrv container with an OpenRV build (the ci-openrv container just has the dependencies for such a build), and add openrv to ci-vfxall/image.yaml to pull your OpenRV build into the all-emcompassing ci-vfxall image.

[jfpanisset]

Although the VFX Platform does not mandate a version of OpenRV, aswf-docker containers try to version everything that isn't coming as RPMs in the base package. So you'll want to add ASWF_OPENRV_VERSION that can point to latest version for now.

[jfpanisset]

The other build_XXX.sh scripts use curl to fetch a versioned tarball, for instance build_osl.sh:

if [ ! -f "$DOWNLOADS_DIR/osl-${ASWF_OSL_VERSION}.tar.gz" ]; then
    curl --location "https://github.com/AcademySoftwareFoundation/OpenShadingLanguage/archive/refs/tags/v${ASWF_OSL_VERSION}.tar.gz" -o "$DOWNLOADS_DIR/osl-${ASWF_OSL_VERSION}.tar.gz"
fi

So you'll want to do something like that based on ASWF_OPENRV_VERSION from versions.yaml.

Also as an effort to improve supply chain security many ASWF projects are working towards having signed source releases, so we're hoping to add signature verification of source tarball downloads.

[jfpanisset]

Are you planning longer term to try to build OpenRV against the PySide2 provided by aswf-docker?

[cedrik-fuoco-adsk]

I'd like to be able to allow to build it from source or to use an external PySide. I think it will be more doable once OpenRV is ported to Qt6/PySide6/Shiboken6 and OpenSSL 3.

However, I don't have a timeline and I can't give an assurance that this will be done right now.

[jfpanisset]

Might be best to be more explicit about where you expect rvsetup to come from, the cwd gets tricky in a CI environment.

[jfpanisset]

Best to use a transformation off ASWF_VFXPLATFORM_VERSION rather than hard coding the year.

[jfpanisset]

This should install in the default /usr/local like the other packages.

[jfpanisset]

Turns out I had forgotten to click on Submit Review, sorry about that.

I did integrate most of the changes in the recently released 2024.2 images, and there's now a ci-openrv:2024.2 build image you can try out.

One issue I ran into is that by default OpenRV seems to expect to build inside a git checkout, for the purposes of including a test build of OpenRV (once that works) it would be nice if it were possible to check out a specific OpenRV version and build that.