Skip to content

MacOS M1 binary compilation and possibility to fine tune a model in C++ #11343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jplu opened this issue Apr 25, 2022 · 8 comments
Open

MacOS M1 binary compilation and possibility to fine tune a model in C++ #11343

jplu opened this issue Apr 25, 2022 · 8 comments
Labels
training issues related to ONNX Runtime training; typically submitted using template

Comments

@jplu
Copy link

jplu commented Apr 25, 2022 •
edited
Loading

Describe the bug
Issue to compile ONNX Runtime with training enabled.

Urgency
Would be perfect to get an answer before the end of the first week of May, means at most the 6th of May. As I have to propose a POC before that date.

System information

  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): MacOS 12.3.1 M1
  • ONNX Runtime installed from (source or binary): none (trying to compile from source)
  • ONNX Runtime version: 1.12.0 (from master branch commit 6fb29f5)
  • Python version: none
  • Visual Studio version (if applicable): none
  • GCC/Compiler version (if compiling from source):
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
  • CUDA/cuDNN version: none
  • GPU model and memory: M1 with 16GB

To Reproduce

  • Describe steps/code to reproduce the behavior:
./build.sh --cmake_extra_defines ONNXRUNTIME_VERSION=`cat ./VERSION_NUMBER` --build_dir build --config Release --parallel --skip_tests --enable_training --use_xcode --osx_arch arm64

Expected behavior
Properly compile ONNX Runtime

Additional context
The compilation error is:

CompileC /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/ruby_generator.o /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/ruby/ruby_generator.cc normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'libprotoc' from project 'onnxruntime')
    cd /Users/jplu/dev/onnxruntime/cmake
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -x c++ -target arm64-apple-macos12.3 -fmessage-length\=545 -fdiagnostics-show-note-include-stack -fmacro-backtrace-limit\=0 -fcolor-diagnostics -Wno-trigraphs -fpascal-strings -O3 -Wno-missing-field-initializers -Wno-missing-prototypes -Wno-return-type -Wno-non-virtual-dtor -Wno-overloaded-virtual -Wno-exit-time-destructors -Wno-missing-braces -Wparentheses -Wswitch -Wno-unused-function -Wno-unused-label -Wno-unused-parameter -Wno-unused-variable -Wunused-value -Wno-empty-body -Wno-uninitialized -Wno-unknown-pragmas -Wno-shadow -Wno-four-char-constants -Wno-conversion -Wno-constant-conversion -Wno-int-conversion -Wno-bool-conversion -Wno-enum-conversion -Wno-float-conversion -Wno-non-literal-null-conversion -Wno-objc-literal-conversion -Wno-shorten-64-to-32 -Wno-newline-eof -Wno-c++11-extensions -DCMAKE_INTDIR\=\"Release\" -DLIBPROTOC_EXPORTS -DEIGEN_MPL2_ONLY -DENABLE_CPU_FP16_TRAINING_OPS -DGOOGLE_PROTOBUF_CMAKE_BUILD -DHAVE_PTHREAD -DHAVE_ZLIB -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fstrict-aliasing -Wdeprecated-declarations -Winvalid-offsetof -Wno-sign-conversion -Wno-infinite-recursion -Wno-move -Wno-comma -Wno-block-capture-autoreleasing -Wno-strict-prototypes -Wno-range-loop-analysis -Wno-semicolon-before-method-body -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release/include -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake -I/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources-normal/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources -F/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release -ffunction-sections -fdata-sections -fvisibility\=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DNDEBUG -fPIC -std\=c++11 -MMD -MT dependencies -MF /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/ruby_generator.d --serialize-diagnostics /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/ruby_generator.dia -c /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/ruby/ruby_generator.cc -o /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/ruby_generator.o
/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/ruby_generator.dia:1:1: warning: Could not read serialized diagnostics file: error("Failed to open diagnostics file") (in target 'libprotoc' from project 'onnxruntime')

CompileC /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/python_generator.o /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/python/python_generator.cc normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'libprotoc' from project 'onnxruntime')
    cd /Users/jplu/dev/onnxruntime/cmake
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -x c++ -target arm64-apple-macos12.3 -fmessage-length\=545 -fdiagnostics-show-note-include-stack -fmacro-backtrace-limit\=0 -fcolor-diagnostics -Wno-trigraphs -fpascal-strings -O3 -Wno-missing-field-initializers -Wno-missing-prototypes -Wno-return-type -Wno-non-virtual-dtor -Wno-overloaded-virtual -Wno-exit-time-destructors -Wno-missing-braces -Wparentheses -Wswitch -Wno-unused-function -Wno-unused-label -Wno-unused-parameter -Wno-unused-variable -Wunused-value -Wno-empty-body -Wno-uninitialized -Wno-unknown-pragmas -Wno-shadow -Wno-four-char-constants -Wno-conversion -Wno-constant-conversion -Wno-int-conversion -Wno-bool-conversion -Wno-enum-conversion -Wno-float-conversion -Wno-non-literal-null-conversion -Wno-objc-literal-conversion -Wno-shorten-64-to-32 -Wno-newline-eof -Wno-c++11-extensions -DCMAKE_INTDIR\=\"Release\" -DLIBPROTOC_EXPORTS -DEIGEN_MPL2_ONLY -DENABLE_CPU_FP16_TRAINING_OPS -DGOOGLE_PROTOBUF_CMAKE_BUILD -DHAVE_PTHREAD -DHAVE_ZLIB -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fstrict-aliasing -Wdeprecated-declarations -Winvalid-offsetof -Wno-sign-conversion -Wno-infinite-recursion -Wno-move -Wno-comma -Wno-block-capture-autoreleasing -Wno-strict-prototypes -Wno-range-loop-analysis -Wno-semicolon-before-method-body -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release/include -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake -I/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources-normal/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources -F/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release -ffunction-sections -fdata-sections -fvisibility\=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DNDEBUG -fPIC -std\=c++11 -MMD -MT dependencies -MF /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/python_generator.d --serialize-diagnostics /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/python_generator.dia -c /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/python/python_generator.cc -o /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/python_generator.o
/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/python_generator.dia:1:1: warning: Could not read serialized diagnostics file: error("Failed to open diagnostics file") (in target 'libprotoc' from project 'onnxruntime')

CompileC /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.pb.o /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/plugin.pb.cc normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'libprotoc' from project 'onnxruntime')
    cd /Users/jplu/dev/onnxruntime/cmake
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -x c++ -target arm64-apple-macos12.3 -fmessage-length\=545 -fdiagnostics-show-note-include-stack -fmacro-backtrace-limit\=0 -fcolor-diagnostics -Wno-trigraphs -fpascal-strings -O3 -Wno-missing-field-initializers -Wno-missing-prototypes -Wno-return-type -Wno-non-virtual-dtor -Wno-overloaded-virtual -Wno-exit-time-destructors -Wno-missing-braces -Wparentheses -Wswitch -Wno-unused-function -Wno-unused-label -Wno-unused-parameter -Wno-unused-variable -Wunused-value -Wno-empty-body -Wno-uninitialized -Wno-unknown-pragmas -Wno-shadow -Wno-four-char-constants -Wno-conversion -Wno-constant-conversion -Wno-int-conversion -Wno-bool-conversion -Wno-enum-conversion -Wno-float-conversion -Wno-non-literal-null-conversion -Wno-objc-literal-conversion -Wno-shorten-64-to-32 -Wno-newline-eof -Wno-c++11-extensions -DCMAKE_INTDIR\=\"Release\" -DLIBPROTOC_EXPORTS -DEIGEN_MPL2_ONLY -DENABLE_CPU_FP16_TRAINING_OPS -DGOOGLE_PROTOBUF_CMAKE_BUILD -DHAVE_PTHREAD -DHAVE_ZLIB -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fstrict-aliasing -Wdeprecated-declarations -Winvalid-offsetof -Wno-sign-conversion -Wno-infinite-recursion -Wno-move -Wno-comma -Wno-block-capture-autoreleasing -Wno-strict-prototypes -Wno-range-loop-analysis -Wno-semicolon-before-method-body -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release/include -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake -I/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources-normal/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources -F/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release -ffunction-sections -fdata-sections -fvisibility\=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DNDEBUG -fPIC -std\=c++11 -MMD -MT dependencies -MF /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.pb.d --serialize-diagnostics /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.pb.dia -c /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/plugin.pb.cc -o /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.pb.o
/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.pb.dia:1:1: warning: Could not read serialized diagnostics file: error("Failed to open diagnostics file") (in target 'libprotoc' from project 'onnxruntime')

CompileC /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.o /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/plugin.cc normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'libprotoc' from project 'onnxruntime')
    cd /Users/jplu/dev/onnxruntime/cmake
    /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang -x c++ -target arm64-apple-macos12.3 -fmessage-length\=545 -fdiagnostics-show-note-include-stack -fmacro-backtrace-limit\=0 -fcolor-diagnostics -Wno-trigraphs -fpascal-strings -O3 -Wno-missing-field-initializers -Wno-missing-prototypes -Wno-return-type -Wno-non-virtual-dtor -Wno-overloaded-virtual -Wno-exit-time-destructors -Wno-missing-braces -Wparentheses -Wswitch -Wno-unused-function -Wno-unused-label -Wno-unused-parameter -Wno-unused-variable -Wunused-value -Wno-empty-body -Wno-uninitialized -Wno-unknown-pragmas -Wno-shadow -Wno-four-char-constants -Wno-conversion -Wno-constant-conversion -Wno-int-conversion -Wno-bool-conversion -Wno-enum-conversion -Wno-float-conversion -Wno-non-literal-null-conversion -Wno-objc-literal-conversion -Wno-shorten-64-to-32 -Wno-newline-eof -Wno-c++11-extensions -DCMAKE_INTDIR\=\"Release\" -DLIBPROTOC_EXPORTS -DEIGEN_MPL2_ONLY -DENABLE_CPU_FP16_TRAINING_OPS -DGOOGLE_PROTOBUF_CMAKE_BUILD -DHAVE_PTHREAD -DHAVE_ZLIB -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -fstrict-aliasing -Wdeprecated-declarations -Winvalid-offsetof -Wno-sign-conversion -Wno-infinite-recursion -Wno-move -Wno-comma -Wno-block-capture-autoreleasing -Wno-strict-prototypes -Wno-range-loop-analysis -Wno-semicolon-before-method-body -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release/include -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake -I/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources-normal/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources/arm64 -I/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/DerivedSources -F/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/Release -ffunction-sections -fdata-sections -fvisibility\=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DNDEBUG -fPIC -std\=c++11 -MMD -MT dependencies -MF /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.d --serialize-diagnostics /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.dia -c /Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/compiler/plugin.cc -o /Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.o
/Users/jplu/dev/onnxruntime/build/Release/external/protobuf/cmake/onnxruntime.build/Release/libprotoc.build/Objects-normal/arm64/plugin.dia:1:1: warning: Could not read serialized diagnostics file: error("Failed to open diagnostics file") (in target 'libprotoc' from project 'onnxruntime')

** BUILD FAILED **


The following build commands failed:
	CompileC /Users/jplu/dev/onnxruntime/build/Release/external/googletest/googletest/onnxruntime.build/Release/gtest.build/Objects-normal/arm64/gtest-all.o /Users/jplu/dev/onnxruntime/cmake/external/googletest/googletest/src/gtest-all.cc normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'gtest' from project 'onnxruntime')
(1 failure)
Traceback (most recent call last):
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 2493, in <module>
    sys.exit(main())
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 2408, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 1245, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 673, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/Users/jplu/dev/onnxruntime/tools/python/util/run.py", line 42, in run
    completed_process = subprocess.run(
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/cmake', '--build', 'build/Release', '--config', 'Release', '--parallel', '8']' returned non-zero exit status 65.

I suspect to do not provide the proper parameters, but there is a lack of documentation on this part, the only thing I found is this page.

This bring me to another question, is training and fine tuning models from C++ with ONNX Runtime possible on MacOS M1 at least? If not it is useless for me to go deeper in this.

Thanks a lot in advance for any light you could provide on this issue.
@snnn
Copy link
Member

snnn commented Apr 25, 2022

I think the argument doesn't work properly. So I created a PR to remove it, but it didn't get enough eye sights.
#8754

And a PR to clarify the build instructions:
#8710

I would suggest you build it with

--cmake_extra_defines CMAKE_OSX_ARCHITECTURES=arm64

Let me know if the error still exists.

@snnn
Copy link
Member

snnn commented Apr 25, 2022

For your information, the macOS binaries in our github release package were built in a way like:

./build.sh  --build --build_java --build_nodejs --use_coreml --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=arm64  --parallel --build_shared_lib --config Release

I think you don't need to use "--build_java --build_nodejs"

And we haven't tried it for training. I know training x86_64 works. I haven't tried arm. I hope there won't be problems.

@jplu
Copy link
Author

jplu commented Apr 26, 2022 •
edited
Loading

Thanks! Now with this command line:

./build.sh  --build_dir build --use_coreml --cmake_extra_defines CMAKE_OSX_ARCHITECTURES=arm64 --skip_tests --enable_training  --parallel --build_shared_lib --config Release

I get a different compilation error but still related to protobuf:

[  9%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/__/src/google/protobuf/wire_format_lite.cc.o
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/stubs/strutil.cc.o
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/stubs/time.cc.o
/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/stubs/strutil.cc:1255:19: error: expected unqualified-id
  } else if (std::isnan(value)) {
                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/math.h:165:5: note: expanded from macro 'isnan'
    ( sizeof(x) == sizeof(float)  ? __inline_isnanf((float)(x))          \
    ^
/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/stubs/strutil.cc:1373:19: error: expected unqualified-id
  } else if (std::isnan(value)) {
                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/math.h:165:5: note: expanded from macro 'isnan'
    ( sizeof(x) == sizeof(float)  ? __inline_isnanf((float)(x))          \
    ^
2 errors generated.
make[2]: *** [external/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/__/src/google/protobuf/stubs/strutil.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/wire_format_lite.cc.o
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/any.cc.o
180 warnings generated.
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/any.pb.cc.o
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/api.pb.cc.o
[ 10%] Building CXX object external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/compiler/importer.cc.o
/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/stubs/strutil.cc:1255:19: error: expected unqualified-id
  } else if (std::isnan(value)) {
                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/math.h:165:5: note: expanded from macro 'isnan'
    ( sizeof(x) == sizeof(float)  ? __inline_isnanf((float)(x))          \
    ^
/Users/jplu/dev/onnxruntime/cmake/external/protobuf/src/google/protobuf/stubs/strutil.cc:1373:19: error: expected unqualified-id
  } else if (std::isnan(value)) {
                  ^
/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/math.h:165:5: note: expanded from macro 'isnan'
    ( sizeof(x) == sizeof(float)  ? __inline_isnanf((float)(x))          \
    ^
2 errors generated.
make[2]: *** [external/protobuf/cmake/CMakeFiles/libprotobuf.dir/__/src/google/protobuf/stubs/strutil.cc.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [external/protobuf/cmake/CMakeFiles/libprotobuf-lite.dir/all] Error 2
make[1]: *** [external/protobuf/cmake/CMakeFiles/libprotobuf.dir/all] Error 2
make: *** [all] Error 2
Traceback (most recent call last):
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 2493, in <module>
    sys.exit(main())
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 2408, in main
    build_targets(args, cmake_path, build_dir, configs, num_parallel_jobs, args.target)
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 1245, in build_targets
    run_subprocess(cmd_args, env=env)
  File "/Users/jplu/dev/onnxruntime/tools/ci_build/build.py", line 673, in run_subprocess
    return run(*args, cwd=cwd, capture_stdout=capture_stdout, shell=shell, env=my_env)
  File "/Users/jplu/dev/onnxruntime/tools/python/util/run.py", line 42, in run
    completed_process = subprocess.run(
  File "/opt/homebrew/Caskroom/miniforge/base/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/opt/homebrew/bin/cmake', '--build', 'build/Release', '--config', 'Release', '--', '-j8']' returned non-zero exit status 2.

@snnn
Copy link
Member

snnn commented Apr 26, 2022

I haven't seen such error before. It looks like the compiler you are using isn't compatible with the protobuf version we are using. When the build just started, cmake should print out compiler information. Would you please post it here? Is it Apple Clang?_

@jplu
Copy link
Author

jplu commented Apr 26, 2022

Here the summary given by CMake:

--
-- ******** Summary ********
--   CMake version             : 3.22.2
--   CMake command             : /opt/homebrew/Cellar/cmake/3.22.2/bin/cmake
--   System                    : Darwin
--   C++ compiler              : /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
--   C++ compiler version      : 13.1.6.13160021
--   CXX flags                 :  -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DCPUINFO_SUPPORTED -Wno-deprecated -Wnon-virtual-dtor
--   Build type                : Release
--   Compile definitions       : EIGEN_MPL2_ONLY;ENABLE_CPU_FP16_TRAINING_OPS;PLATFORM_POSIX;__STDC_FORMAT_MACROS
--   CMAKE_PREFIX_PATH         :
--   CMAKE_INSTALL_PREFIX      : /usr/local
--   CMAKE_MODULE_PATH         : /Users/jplu/dev/onnxruntime/cmake/external
--
--   ONNX version              : 1.11.0
--   ONNX NAMESPACE            : onnx
--   ONNX_USE_LITE_PROTO       : ON
--   USE_PROTOBUF_SHARED_LIBS  : OFF
--   Protobuf_USE_STATIC_LIBS  : ON
--   ONNX_DISABLE_EXCEPTIONS   : OFF
--   ONNX_WERROR               : OFF
--   ONNX_BUILD_TESTS          : OFF
--   ONNX_BUILD_BENCHMARKS     : OFF
--   ONNXIFI_DUMMY_BACKEND     : OFF
--   ONNXIFI_ENABLE_EXT        : OFF
--
--   Protobuf compiler         :
--   Protobuf includes         :
--   Protobuf libraries        :
--   BUILD_ONNX_PYTHON         : OFF
Use flatbuffers from submodule
-- Checking for module 'mpi-c'
--   No package 'mpi-c' found
-- Could NOT find MPI_C (missing: MPI_C_LIB_NAMES MPI_C_HEADER_DIR MPI_C_WORKS)
-- Checking for module 'mpi-cxx'
--   No package 'mpi-cxx' found
-- Could NOT find MPI_CXX (missing: MPI_CXX_LIB_NAMES MPI_CXX_HEADER_DIR MPI_CXX_WORKS)
-- Could NOT find MPI (missing: MPI_C_FOUND MPI_CXX_FOUND)
CMake Warning at CMakeLists.txt:1843 (message):
  MPI is not found.  Please define onnxruntime_MPI_HOME to specify the path
  of MPI.  Otherwise, NCCL will be disabled.


-- Configuring done
-- Generating done

And the compiler version that CMake uses:

/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ --version
Apple clang version 13.1.6 (clang-1316.0.21.2.3)
Target: arm64-apple-darwin21.4.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

@snnn
Copy link
Member

snnn commented Apr 26, 2022

And for your information, this is what we have now:

--   CMake command             : /usr/local/Cellar/cmake/3.23.1/bin/cmake
--   System                    : Darwin
--   C++ compiler              : /Applications/Xcode_12.4.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++
--   C++ compiler version      : 12.0.0.12000032
--   CXX flags                 :  -ffunction-sections -fdata-sections -fvisibility=hidden -fvisibility-inlines-hidden -fstack-protector-strong -DCPUINFO_SUPPORTED -Wno-deprecated -Wnon-virtual-dtor
--   Build type                : Release
--   Compile definitions       : EIGEN_MPL2_ONLY;PLATFORM_POSIX;__STDC_FORMAT_MACROS
--   CMAKE_PREFIX_PATH         : 
--   CMAKE_INSTALL_PREFIX      : /usr/local
--   CMAKE_MODULE_PATH         : /Users/runner/work/1/s/cmake/external
-- 
--   ONNX version              : 1.11.0
--   ONNX NAMESPACE            : onnx
--   ONNX_USE_LITE_PROTO       : ON
--   USE_PROTOBUF_SHARED_LIBS  : OFF
--   Protobuf_USE_STATIC_LIBS  : ON
--   ONNX_DISABLE_EXCEPTIONS   : OFF
--   ONNX_WERROR               : OFF
--   ONNX_BUILD_TESTS          : OFF
--   ONNX_BUILD_BENCHMARKS     : OFF
--   ONNXIFI_DUMMY_BACKEND     : OFF
--   ONNXIFI_ENABLE_EXT        : OFF
-- 
--   Protobuf compiler         : 
--   Protobuf includes         : 
--   Protobuf libraries        : 
--   BUILD_ONNX_PYTHON         : OFF

Yours is newer. I will try to see if I can get it one, otherwise it's difficult for me to work on.

@jplu
Copy link
Author

jplu commented Apr 26, 2022

My version of Xcode is 13.3.1 and I'm on MacOS Monterey 12.3.1 if it may help you.

@jplu
Copy link
Author

jplu commented May 4, 2022

@snnn Any update on your side on testing the compilation on a newer environment?

@manashgoswami manashgoswami added the training issues related to ONNX Runtime training; typically submitted using template label May 9, 2022
@PickledChair PickledChair mentioned this issue Feb 17, 2023
Open
3 tasks
@csukuangfj csukuangfj mentioned this issue May 12, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
training issues related to ONNX Runtime training; typically submitted using template
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@snnn @jplu @manashgoswami