Python MacOS arm64 release binaries

[lutzroeder]

Describe the bug

ONNX Runtime does not install using pip on M1.

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): macOS 11.2.1
• ONNX Runtime installed from (source or binary): pip
• Python version: 3.9.1

To Reproduce

~: uname -v
Darwin Kernel Version 20.3.0: Thu Jan 21 00:06:51 PST 2021; root:xnu-7195.81.3~1/RELEASE_ARM64_T8101
~: which python3
/opt/homebrew/bin/python3
~: which pip
/opt/homebrew/bin/pip
~: python3 --version
Python 3.9.1
~: pip install onnxruntime
ERROR: Could not find a version that satisfies the requirement onnxruntime
ERROR: No matching distribution found for onnxruntime

[snnn]

What kind of tag we should publish? Does pypi have a standard for that?

[lutzroeder]

@ronaldoussoren is there some official guidance? Guessing that macosx_v_universal2 would be more convenient vs. macosx_v_arm64 and macosx_v_x64_86? See also pypa/packaging#319.

[snnn]

Is it in the latest pip release? What's the version requirement of pip to support it?

[ronaldoussoren]

pip 21.0.1 can install universal2 wheels (I've just verified this on my machine). For my own projects I currently publish an x86_64 wheel as before and a universal2 wheel for folks with M1 systems. I intend to drop the x86_64 wheels from releases later this year.

My personal preference is to have universal2 wheels. That reduces the amount of different wheels needed, and more importantly makes it a lot easier to build applications that can be used on both Intel and M1 Macs using tools like py2app and pyinstaller.

For projects with huge wheels such as tensorflow (which wheels well over 150MB) having per-architecture wheels is likely better due to disk usage considerations.

[snnn]

@ronaldoussoren Thank you for the information, it is really helpful. We will look into it.

Currently we are preparing the release for 1.7. The deadline is soon. Likely this feature won't be in that release. Sorry.

[lutzroeder]

onnx/onnx#3129

[sarim-zafar]

any update on this?

[snnn]

I'm working on it.

[snnn]

Reference:

1. https://developer.apple.com/documentation/apple-silicon/building-a-universal-macos-binary
2. add support for macos 11.0, arm64, universal2 pypa/packaging#319

I think we can do cross-compiling on a Mac X86 hardware. Anybody knows more details? What should I do?

[snnn]

Let me try to leverage https://cibuildwheel.readthedocs.io/en/stable/options/

[hossein1387]

what is the plan to support Onnxruntime on M1 chip?

[OlegTheCat]

Being able to run ONNX on M1 Macs is very desired. Currently, there's no ability to do development with ONNX on these laptops. So instead, I have to use a virtual machine or rent a cloud x64 instance, which impedes productivity or creates additional costs.

[snnn]

I will submit a PR and publish a test package tomorrow.

[snnn]

https://test.pypi.org/project/onnxruntime/1.8.2.dev20210816004/
To install it:

pip3 install --upgrade pip
pip3 install numpy
pip3 install -i https://test.pypi.org/simple/ onnxruntime==1.8.2.dev20210816004

[hossein1387]

I followed your instructions and I am getting the following error:

>>> import onnxruntime
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/hossein/miniforge3/lib/python3.9/site-packages/onnxruntime/__init__.py", line 34, in <module>
    raise import_capi_exception
  File "/Users/hossein/miniforge3/lib/python3.9/site-packages/onnxruntime/__init__.py", line 23, in <module>
    from onnxruntime.capi._pybind_state import get_all_providers, get_available_providers, get_device, set_seed, \
  File "/Users/hossein/miniforge3/lib/python3.9/site-packages/onnxruntime/capi/_pybind_state.py", line 21, in <module>
    from .onnxruntime_pybind11_state import *  # noqa
ImportError: dlopen(/Users/hossein/miniforge3/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so, 2): no suitable image found.  Did find:
	/Users/hossein/miniforge3/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so: mach-o, but wrong architecture
	/Users/hossein/miniforge3/lib/python3.9/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so: mach-o, but wrong architecture

[snnn]

What's your pip version?

[hossein1387]

->pip --version
pip 21.2.4 from /Users/hossein/miniforge3/lib/python3.9/site-packages/pip (python 3.9)

[snnn]

Looks like something is wrong. Let me debug it.

[snnn]

I found the problem. Fixing.

[snnn]

It needs quite a lot code changes. CMAKE_OSX_ARCHITECTURES is a list, we shouldn't use it in a way like:

if (CMAKE_OSX_ARCHITECTURES STREQUAL "arm64")

See: #3298

[snnn]

Now I got a new one, please try:
https://test.pypi.org/project/ort-nightly/1.8.2.dev169944/

Currently it only has python 3.8 and 3.9. This is because only CPython 3.8 and newer support universal2 and arm64 wheels.

It was built from commit id: 617f0f5

[hossein1387]

Works perfectly! Thanks a lot!

[unbeatable-101]

I'm getting

ERROR: Could not find a version that satisfies the requirement numpy>=1.16.6 (from ort-nightly) (from versions: 1.9.3)
ERROR: No matching distribution found for numpy>=1.16.6

I have numpy-1.21.2 installed

[Fohlen]

Hey @snn when can we expect this to be regularly available at least in the nightly? (right now only very specific commits have the arm64)

[OlegTheCat]

Hi @snnn! It seems that universal binary (or at least a separate binary for osx-aarch64) has not been added to the Java packages in this (1.10) release:

$ jar xf ~/.m2/repository/com/microsoft/onnxruntime/onnxruntime/1.10.0/onnxruntime-1.10.0.jar

$ ls ai/onnxruntime/native
linux-x64 osx-x64   win-x64

$ lipo -archs ai/onnxruntime/native/osx-x64/libonnxruntime.dylib
x86_64

Is it something that has been missed?

[snnn]

Yes, I should update https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/windows/jar_packaging.ps1.

[lucasjinreal]

@snnn Does onnxruntime m1 version ready to publish to pypi?

[snnn]

Not yet.

[lucasjinreal]

@snnn I found ARM build version on my own side even slower than x86 version (Conda x86 python vs Miniforge amr64 python).

I don't know why

[aressem]

Any ETA when linux-aarch64 / osx-aarch64 can be included in a released Java (.jar) version of onnxruntime ?

[ZetiMente]

https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/windows/jar_packaging.ps1

Hi, it's now one month later. Is there any chance this will be soon? Please. 😀

[lucasjinreal]

Hi, first quarter of 2022 are coming to end, is there any updates on arm M1 version?

[XavierGeerinck]

I would also love to see an updated ARM M1 version on PyPi

[damienrj]

Same, a lot of errors trying to install on m1.

[NiuGeng-SEU]

Same, cannot find a way to deal with it.

[lucasjinreal]

Hi, 100 years later, any m1 version onnxruntime? If not, how to build on my self? I don't know what's the command line to build with cmake enable arm64

[snnn]

See: https://github.com/microsoft/onnxruntime/pull/8710/files

f you want to cross-compile for Apple Silicon in an Intel-based MacOS machine, please add the argument --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=arm64 with cmake > 3.19

if you want to generate a Universal2 Binary, please use --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=x86_64;arm64

[Fohlen]

Hi, 100 years later, any m1 version onnxruntime? If not, how to build on my self? I don't know what's the command line to build with cmake enable arm64

Please bear in mind that OSS in general, and ONNX in particular, is a project run by volunteers (in this case Microsoft volunteering their resources) and is available freely to the world. Smirky comments like '100 years later' are not only rude, they will discourage others from contributing, essentially harming everyone including yourself. Please refrain from such an attitude in the future.

That being said @snnn there are working wheels on the nightly version of this repository. What would be necessary to include them in ALL of the future nightly releases? Can we help by adding a PR? Once the coverage is sufficient it can then go from the nightly to stable.. what's preventing this from happening? A little insight here would be much appreciated

[snnn]

Thanks for the understanding. I just had a discussion with our product team. The feature will be included in ONNX Runtime 1.12 release. @mszhanyi will work on it. He will update the "MacOS_py_Wheels" build job in https://github.com/microsoft/onnxruntime/blob/master/tools/ci_build/github/azure-pipelines/templates/py-packaging-stage.yml to include arm64. We should provide two types of wheels: one for x86_64, another for universal2.

[snnn]

@faxu , would you please create a 1.12 project at https://github.com/microsoft/onnxruntime/projects ?

[faxu]

https://github.com/microsoft/onnxruntime/projects/9

Will populate soon with discussed release plans.

[andreaferretti]

@snnn The packages you linked do not exist anymore, is there another way to install onnxruntime? Or maybe could you please provide updated packages?

[cansik]

If you are interested in pre-built onnxruntime wheels for Mac M1 (Apple Silicon), I have created a replacement package called onnxruntime-silicon.

pip install onnxruntime-silicon

[andreaferretti]

@cansik Great, works on the first try, thank you! :-)

[fxmarty]

@cansik Out of curiosity, do you have any idea if this can be leveraged for quantization on Apple M1 chip? Or there is still an execution provider missing for this kind of hardware? Thanks!