Track conda-forge migrations with automated tooling

[bdice]

Problem Statement

Today, RAPIDS is not built with conda-forge infrastructure, but aims to be compatible with the conda-forge ecosystem.

There is an extensive set of conda-forge automated bots that perform migration work (for example, most of the rebuilds of libarrow are from migrators). Today, RAPIDS does not benefit from conda-forge's automation for version migrations, yet RAPIDS must track conda-forge migrations in order to be compatible with the latest conda-forge packages.

Tracking conda-forge migrations in RAPIDS has historically been a pain point. We've had conflicts with spdlog and fmt numerous times, and updating those pinnings has been difficult to do smoothly (typically because of patches needed in the upstream repos). We used to have similar issues with Arrow, though libcudf has recently dropped the build requirement on libarrow (at least in part to ease this pain).

See also:

• This discussion on adding an aws-sdk-cpp dependency to libkvikio: S3 support kvikio#426 (comment)
• Keith's warning that RAPIDS is lagging conda-forge in spdlog/fmt and should expect issues: update fmt and spdlog across RAPIDS #56 (comment)
• This problem in cuspatial that came from spdlog/fmt being behind: Add support for Python 3.12, pin proj back to 9.3.x, require geopandas>=1.0 cuspatial#1453 (comment)

We need to improve our integration with the conda-forge ecosystem by following conda-forge migrations in RAPIDS builds.

Possible solutions

We could implement similar automation to what conda-forge uses, with various degrees of complexity. We might be happy with plain notifications when a migration is needed on our end. The following are not mutually exclusive. I am open to any possible improvements that fix problems (ideally with some automation) and/or give us more information about conflicts or work to be done.

Solution 1: conda_build_config.yaml alignment

Use https://github.com/conda-forge/conda-forge-pinning-feedstock/blob/main/recipe/conda_build_config.yaml as a "base version set" for RAPIDS conda builds. This would make RAPIDS match the state of conda-forge's completed migrations. This would keep RAPIDS up to date, but could be painful if the rollover breaks CI. We could issue our own mini-migrations that keep RAPIDS aligned with rapids-reviser, perhaps.

Solution 2: notifications of migrations

Set up a system that tracks a few core dependencies that we flag (e.g. spdlog, fmt, aws-sdk-cpp, libarrow), watches for migrations that touch those, and files task issues on a central RAPIDS repository.

Solution 3: notifications of conflicts with "latest"

Set up a system that notifies us when RAPIDS nightlies cannot be used with the latest versions of conda-forge dependencies. This could look something like rapidsai/integration#690, with a few tweaks.