Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CI] Stop vendoring libomp.dylib in MacOS Python wheels #10440

Merged
merged 11 commits into from
Jun 19, 2024

Conversation

hcho3
Copy link
Collaborator

@hcho3 hcho3 commented Jun 18, 2024

Adopt microsoft/LightGBM#6391 to achieve the following goals:

  • Stop vendoring libomp in the MacOS Python wheels. Currently, we vendor libomp 11.1.0, a 3-year old version.
  • Allow XGBoost to work with libomp from a variety of sources, e.g. Conda or Homebrew.
    • libxgboost.dylib no longer has dependency on hardcoded path of libomp.dylib. So it can be used in conjunction with any installation of libomp.dylib.
    • If a copy of libomp.dylib is already present in memory, XGBoost will use that copy. This way, we can avoid myriad problems caused by having multiple incompatible versions of libomp (Segmentation fault on MacOS with libomp 12.0.0 #7039).

One disadvantage of this approach is that users will now get an OSError if OpenMP is not present in the system. This is not too bad, since the error message already directs users to install OpenMP:

XGBoost Library ({libname}) could not be loaded.
Likely causes:
* OpenMP runtime is not installed
- vcomp140.dll or libgomp-1.dll for Windows
- libomp.dylib for Mac OSX
- libgomp.so for Linux and other UNIX-like OSes
Mac OSX users: Run `brew install libomp` to install OpenMP runtime.

Before this commit (note the hard-coded path for libomp.dylib):

$ otool -L ./lib/libxgboost.dylib
./lib/libxgboost.dylib:
        @rpath/libxgboost.dylib (compatibility version 0.0.0, current version 0.0.0)
        /opt/homebrew/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

After this commit:

$ otool -L ./lib/libxgboost.dylib
./lib/libxgboost.dylib:
        @rpath/libxgboost.dylib (compatibility version 0.0.0, current version 0.0.0)
        @rpath/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
        /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

@jameslamb Thanks for inspiration!

@hcho3 hcho3 marked this pull request as draft June 18, 2024 01:42
@hcho3 hcho3 added the Blocking label Jun 18, 2024
@hcho3
Copy link
Collaborator Author

hcho3 commented Jun 18, 2024

TODO: Update CI to build libxgboost.dylib without bundling libomp.

@trivialfis trivialfis mentioned this pull request Jun 18, 2024
14 tasks
@hcho3 hcho3 marked this pull request as ready for review June 19, 2024 00:53
@hcho3 hcho3 merged commit 8689f0b into dmlc:master Jun 19, 2024
26 of 29 checks passed
@hcho3 hcho3 deleted the fix_macos_openmp branch June 19, 2024 02:17
hcho3 added a commit to hcho3/xgboost that referenced this pull request Jun 19, 2024
hcho3 added a commit that referenced this pull request Jun 19, 2024
)

* [CI] Stop vendoring libomp.dylib in MacOS Python wheels (#10440)

* [CI] [jvm-packages] Build libxgboost4j.dylib on M1 MacOS with OpenMP support (#10449)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant