diff --git a/doc/changes/v2.1.0.rst b/doc/changes/v2.1.0.rst index 199a427c17ca..84e6419e9ba8 100644 --- a/doc/changes/v2.1.0.rst +++ b/doc/changes/v2.1.0.rst @@ -34,6 +34,27 @@ NCCL is now fetched from PyPI In the previous version, XGBoost statically linked NCCL, which significantly increased the binary size and led to hitting the PyPI repository limit. With the new release, we have made a significant improvement. The new release can now dynamically load NCCL from an external source, reducing the binary size. For the PyPI package, the ``nvidia-nccl-cu12`` package will be fetched during installation. With more downstream packages reusing NCCL, we expect the user environments to be slimmer in the future as well. (#9796, #9804, #10447) +*************************************************** +Parts of the Python package now require glibc 2.28+ +*************************************************** +Starting from 2.1.0, XGBoost Python package will be distributed in two variants: + +* ``manylinux_2_28``: for recent Linux distros with glibc 2.28 or newer. This variant comes with all features enabled. +* ``manylinux2014``: for old Linux distros with glibc older than 2.28. This variant does not support GPU algorithms or federated learning. + +The ``pip`` package manager will automatically choose the correct variant depending on your system. + +Starting from **May 31, 2025**, we will stop distributing the ``manylinux2014`` variant and exclusively +distribute the ``manylinux_2_28`` variant. We made this decision so that our CI/CD pipeline won't have +depend on software components that reached end-of-life (such as CentOS 7). We strongly encourage +everyone to migrate to recent Linux distros in order to use future versions of XGBoost. + +Note. If you want to use GPU algorithms or federated learning on an older Linux distro, you have +two alternatives: + +1. Upgrade to a recent Linux distro with glibc 2.28+. OR +2. Build XGBoost from the source. + ************ Multi-output ************ @@ -98,7 +119,7 @@ This section lists some new features that are general to all language bindings. - Change default metric for gamma regression to ``deviance``. (#9757) - Normalization for learning to rank is now optional with the introduction of the new ``lambdarank_normalization`` parameter. (#10094) - Contribution prediction with ``QuantileDMatrix`` on CPU. (#10043) -- XGBoost on macos no longer bundles OpenMP runtime. Users can install the latest runtime from their dependency manager of choice. (#10440). Along with which, JVM packages on MacoOS are not built with OpenMP support (#10449). +- XGBoost on macos no longer bundles OpenMP runtime. Users can install the latest runtime from their dependency manager of choice. (#10440). Along with which, JVM packages on MacoOS are now built with OpenMP support (#10449). ********* Bug fixes diff --git a/doc/install.rst b/doc/install.rst index bf90a913bd31..e5229702e16c 100644 --- a/doc/install.rst +++ b/doc/install.rst @@ -27,6 +27,26 @@ Pre-built binary wheels are uploaded to PyPI (Python Package Index) for each rel You might need to run the command with ``--user`` flag or use ``virtualenv`` if you run into permission errors. +.. note:: Parts of the Python package now require glibc 2.28+ + + Starting from 2.1.0, XGBoost Python package will be distributed in two variants: + + * ``manylinux_2_28``: for recent Linux distros with glibc 2.28 or newer. This variant comes with all features enabled. + * ``manylinux2014``: for old Linux distros with glibc older than 2.28. This variant does not support GPU algorithms or federated learning. + + The ``pip`` package manager will automatically choose the correct variant depending on your system. + + Starting from **May 31, 2025**, we will stop distributing the ``manylinux2014`` variant and exclusively + distribute the ``manylinux_2_28`` variant. We made this decision so that our CI/CD pipeline won't have + depend on software components that reached end-of-life (such as CentOS 7). We strongly encourage + everyone to migrate to recent Linux distros in order to use future versions of XGBoost. + + Note. If you want to use GPU algorithms or federated learning on an older Linux distro, you have + two alternatives: + + 1. Upgrade to a recent Linux distro with glibc 2.28+. OR + 2. Build XGBoost from the source. + .. note:: Windows users need to install Visual C++ Redistributable XGBoost requires DLLs from `Visual C++ Redistributable diff --git a/tests/buildkite/build-cuda-with-rmm.sh b/tests/buildkite/build-cuda-with-rmm.sh index 42a1edc7947e..126bc6b3f28b 100755 --- a/tests/buildkite/build-cuda-with-rmm.sh +++ b/tests/buildkite/build-cuda-with-rmm.sh @@ -42,7 +42,7 @@ $command_wrapper python tests/ci_build/rename_whl.py \ --platform-tag ${WHEEL_TAG} echo "--- Audit binary wheel to ensure it's compliant with ${WHEEL_TAG} standard" -tests/ci_build/ci_build.sh auditwheel_x86_64 auditwheel repair \ +tests/ci_build/ci_build.sh manylinux_2_28_x86_64 auditwheel repair \ --plat ${WHEEL_TAG} python-package/dist/*.whl $command_wrapper python tests/ci_build/rename_whl.py \ --wheel-path wheelhouse/*.whl \ @@ -50,7 +50,7 @@ $command_wrapper python tests/ci_build/rename_whl.py \ --platform-tag ${WHEEL_TAG} mv -v wheelhouse/*.whl python-package/dist/ # Make sure that libgomp.so is vendored in the wheel -tests/ci_build/ci_build.sh auditwheel_x86_64 bash -c \ +tests/ci_build/ci_build.sh manylinux_2_28_x86_64 bash -c \ "unzip -l python-package/dist/*.whl | grep libgomp || exit -1" echo "--- Upload Python wheel" diff --git a/tests/buildkite/build-cuda.sh b/tests/buildkite/build-cuda.sh index 2ddf9fefe07b..03d2cc8a6a24 100755 --- a/tests/buildkite/build-cuda.sh +++ b/tests/buildkite/build-cuda.sh @@ -41,7 +41,7 @@ $command_wrapper python tests/ci_build/rename_whl.py \ --platform-tag ${WHEEL_TAG} echo "--- Audit binary wheel to ensure it's compliant with ${WHEEL_TAG} standard" -tests/ci_build/ci_build.sh auditwheel_x86_64 auditwheel repair \ +tests/ci_build/ci_build.sh manylinux_2_28_x86_64 auditwheel repair \ --plat ${WHEEL_TAG} python-package/dist/*.whl $command_wrapper python tests/ci_build/rename_whl.py \ --wheel-path wheelhouse/*.whl \ @@ -49,7 +49,7 @@ $command_wrapper python tests/ci_build/rename_whl.py \ --platform-tag ${WHEEL_TAG} mv -v wheelhouse/*.whl python-package/dist/ # Make sure that libgomp.so is vendored in the wheel -tests/ci_build/ci_build.sh auditwheel_x86_64 bash -c \ +tests/ci_build/ci_build.sh manylinux_2_28_x86_64 bash -c \ "unzip -l python-package/dist/*.whl | grep libgomp || exit -1" echo "--- Upload Python wheel" diff --git a/tests/buildkite/build-manylinux2014-aarch64.sh b/tests/buildkite/build-manylinux2014-aarch64.sh new file mode 100644 index 000000000000..fc8d20b7c11f --- /dev/null +++ b/tests/buildkite/build-manylinux2014-aarch64.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +set -euo pipefail + +source tests/buildkite/conftest.sh + +WHEEL_TAG=manylinux2014_aarch64 +command_wrapper="tests/ci_build/ci_build.sh manylinux2014_aarch64" +python_bin="/opt/python/cp310-cp310/bin/python" + +echo "--- Build binary wheel for ${WHEEL_TAG}" +# Patch to add warning about manylinux2014 variant +patch -p0 < tests/buildkite/manylinux2014_warning.patch +$command_wrapper bash -c \ + "cd python-package && ${python_bin} -m pip wheel --no-deps -vvv . --wheel-dir dist/" +git checkout python-package/xgboost/core.py # discard the patch + +$command_wrapper auditwheel repair --plat ${WHEEL_TAG} python-package/dist/*.whl +$command_wrapper ${python_bin} tests/ci_build/rename_whl.py \ + --wheel-path wheelhouse/*.whl \ + --commit-hash ${BUILDKITE_COMMIT} \ + --platform-tag ${WHEEL_TAG} +rm -rf python-package/dist/ +mkdir python-package/dist/ +mv -v wheelhouse/*.whl python-package/dist/ + +echo "--- Upload Python wheel" +buildkite-agent artifact upload python-package/dist/*.whl +$command_wrapper bash -c "${python_bin} -m pip install awscli" +if [[ ($is_pull_request == 0) && ($is_release_branch == 1) ]] +then + ${python_bin} -m awscli s3 \ + cp python-package/dist/*.whl s3://xgboost-nightly-builds/${BRANCH_NAME}/ \ + --acl public-read --no-progress +fi diff --git a/tests/buildkite/build-manylinux2014-x86_64.sh b/tests/buildkite/build-manylinux2014-x86_64.sh new file mode 100644 index 000000000000..3bcc4e7cb20d --- /dev/null +++ b/tests/buildkite/build-manylinux2014-x86_64.sh @@ -0,0 +1,35 @@ +#!/bin/bash + +set -euo pipefail + +source tests/buildkite/conftest.sh + +WHEEL_TAG=manylinux2014_x86_64 +command_wrapper="tests/ci_build/ci_build.sh manylinux2014_x86_64" +python_bin="/opt/python/cp310-cp310/bin/python" + +echo "--- Build binary wheel for ${WHEEL_TAG}" +# Patch to add warning about manylinux2014 variant +patch -p0 < tests/buildkite/manylinux2014_warning.patch +$command_wrapper bash -c \ + "cd python-package && ${python_bin} -m pip wheel --no-deps -vvv . --wheel-dir dist/" +git checkout python-package/xgboost/core.py # discard the patch + +$command_wrapper auditwheel repair --plat ${WHEEL_TAG} python-package/dist/*.whl +$command_wrapper ${python_bin} tests/ci_build/rename_whl.py \ + --wheel-path wheelhouse/*.whl \ + --commit-hash ${BUILDKITE_COMMIT} \ + --platform-tag ${WHEEL_TAG} +rm -rf python-package/dist/ +mkdir python-package/dist/ +mv -v wheelhouse/*.whl python-package/dist/ + +echo "--- Upload Python wheel" +buildkite-agent artifact upload python-package/dist/*.whl +$command_wrapper bash -c "${python_bin} -m pip install awscli" +if [[ ($is_pull_request == 0) && ($is_release_branch == 1) ]] +then + ${python_bin} -m awscli s3 \ + cp python-package/dist/*.whl s3://xgboost-nightly-builds/${BRANCH_NAME}/ \ + --acl public-read --no-progress +fi diff --git a/tests/buildkite/manylinux2014_warning.patch b/tests/buildkite/manylinux2014_warning.patch new file mode 100644 index 000000000000..692a92672d2f --- /dev/null +++ b/tests/buildkite/manylinux2014_warning.patch @@ -0,0 +1,37 @@ +diff --git python-package/pyproject.toml python-package/pyproject.toml +index a273d8c13..dee49686a 100644 +--- python-package/pyproject.toml ++++ python-package/pyproject.toml +@@ -30,8 +30,7 @@ classifiers = [ + ] + dependencies = [ + "numpy", +- "scipy", +- "nvidia-nccl-cu12 ; platform_system == 'Linux' and platform_machine != 'aarch64'" ++ "scipy" + ] + + [project.urls] +diff --git python-package/xgboost/core.py python-package/xgboost/core.py +index e8bc735e6..030972ef2 100644 +--- python-package/xgboost/core.py ++++ python-package/xgboost/core.py +@@ -262,6 +262,18 @@ Likely cause: + ) + raise ValueError(msg) + ++ warnings.warn( ++ "Your system has an old version of glibc (< 2.28). We will stop supporting " ++ "Linux distros with glibc older than 2.28 after **May 31, 2025**. " ++ "Please upgrade to a recent Linux distro (with glibc 2.28+) to use " ++ "future versions of XGBoost.\n" ++ "Note: You have installed the 'manylinux2014' variant of XGBoost. Certain " ++ "features such as GPU algorithms or federated learning are not available. " ++ "To use these features, please upgrade to a recent Linux distro with glibc " ++ "2.28+, and install the 'manylinux_2_28' variant.", ++ FutureWarning ++ ) ++ + return lib + + diff --git a/tests/buildkite/pipeline.yml b/tests/buildkite/pipeline.yml index 4bcfe03f96ea..73a60517a672 100644 --- a/tests/buildkite/pipeline.yml +++ b/tests/buildkite/pipeline.yml @@ -26,18 +26,17 @@ steps: key: run-clang-tidy agents: queue: linux-amd64-cpu - - wait - label: ":console: Build CPU" command: "tests/buildkite/build-cpu.sh" key: build-cpu agents: queue: linux-amd64-cpu - - label: ":console: Build CPU ARM64" + - label: ":console: Build CPU ARM64 + manylinux_2_28_aarch64 wheel" command: "tests/buildkite/build-cpu-arm64.sh" key: build-cpu-arm64 agents: queue: linux-arm64-cpu - - label: ":console: Build CUDA" + - label: ":console: Build CUDA + manylinux_2_28_x86_64 wheel" command: "tests/buildkite/build-cuda.sh" key: build-cuda agents: @@ -62,6 +61,16 @@ steps: key: build-jvm-doc agents: queue: linux-amd64-cpu + - label: ":console: Build manylinux2014_x86_64 wheel" + command: "tests/buildkite/build-manylinux2014-x86_64.sh" + key: build-manylinux2014-x86_64 + agents: + queue: linux-amd64-cpu + - label: ":console: Build manylinux2014_aarch64 wheel" + command: "tests/buildkite/build-manylinux2014-aarch64.sh" + key: build-manylinux2014-aarch64 + agents: + queue: linux-arm64-cpu - wait #### -------- TEST -------- - label: ":console: Test Python package, CPU" diff --git a/tests/ci_build/Dockerfile.manylinux2014_aarch64 b/tests/ci_build/Dockerfile.manylinux2014_aarch64 new file mode 100644 index 000000000000..05a1590eca76 --- /dev/null +++ b/tests/ci_build/Dockerfile.manylinux2014_aarch64 @@ -0,0 +1,15 @@ +FROM quay.io/pypa/manylinux2014_aarch64 + +# Install lightweight sudo (not bound to TTY) +ENV GOSU_VERSION 1.10 +RUN set -ex; \ + curl -o /usr/local/bin/gosu -L "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-arm64" && \ + chmod +x /usr/local/bin/gosu && \ + gosu nobody true + +# Default entry-point to use if running locally +# It will preserve attributes of created files +COPY entrypoint.sh /scripts/ + +WORKDIR /workspace +ENTRYPOINT ["/scripts/entrypoint.sh"] diff --git a/tests/ci_build/Dockerfile.manylinux2014_x86_64 b/tests/ci_build/Dockerfile.manylinux2014_x86_64 new file mode 100644 index 000000000000..6c2150149fe5 --- /dev/null +++ b/tests/ci_build/Dockerfile.manylinux2014_x86_64 @@ -0,0 +1,15 @@ +FROM quay.io/pypa/manylinux2014_x86_64 + +# Install lightweight sudo (not bound to TTY) +ENV GOSU_VERSION 1.10 +RUN set -ex; \ + curl -o /usr/local/bin/gosu -L "https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-amd64" && \ + chmod +x /usr/local/bin/gosu && \ + gosu nobody true + +# Default entry-point to use if running locally +# It will preserve attributes of created files +COPY entrypoint.sh /scripts/ + +WORKDIR /workspace +ENTRYPOINT ["/scripts/entrypoint.sh"] diff --git a/tests/ci_build/Dockerfile.auditwheel_x86_64 b/tests/ci_build/Dockerfile.manylinux_2_28_x86_64 similarity index 100% rename from tests/ci_build/Dockerfile.auditwheel_x86_64 rename to tests/ci_build/Dockerfile.manylinux_2_28_x86_64