LLVM: Clang broken on Darwin; fails to compile code outside of the sandbox

[maleadt]

❯ /opt/homebrew/Cellar/llvm@15/15.0.7/bin/clang -x c -c - << EOD
    #include <math.h>
EOD

❯ /Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/tools/clang -x c -c - << EOD
    #include <math.h>
EOD
<stdin>:1:14: fatal error: 'math.h' file not found
    #include <math.h>
             ^~~~~~~~
1 error generated.

With verbosity cranked up:

❯ /opt/homebrew/Cellar/llvm@15/15.0.7/bin/clang -v -x c -c - << EOD
    #include <math.h>
EOD
Homebrew clang version 15.0.7
Target: arm64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /opt/homebrew/Cellar/llvm@15/15.0.7/bin
 (in-process)
 "/opt/homebrew/Cellar/llvm@15/15.0.7/bin/clang-15" -cc1 -triple arm64-apple-macosx14.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -funwind-tables=2 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu apple-m1 -target-feature +v8.5a -target-feature +crc -target-feature +lse -target-feature +rdm -target-feature +crypto -target-feature +dotprod -target-feature +fp-armv8 -target-feature +neon -target-feature +fp16fml -target-feature +ras -target-feature +rcpc -target-feature +zcm -target-feature +zcz -target-feature +fullfp16 -target-feature +sm4 -target-feature +sha3 -target-feature +sha2 -target-feature +aes -target-abi darwinpcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=lldb -target-linker-version 1022.1 -v -fcoverage-compilation-dir=/Users/tim/Julia/src/pocl/build/lib/kernel/host -resource-dir /opt/homebrew/Cellar/llvm@15/15.0.7/lib/clang/15.0.7 -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk -internal-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/local/include -internal-isystem /opt/homebrew/Cellar/llvm@15/15.0.7/lib/clang/15.0.7/include -internal-externc-isystem /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include -fdebug-compilation-dir=/Users/tim/Julia/src/pocl/build/lib/kernel/host -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -fcolor-diagnostics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o -.o -x c -
clang -cc1 version 15.0.7 based upon LLVM 15.0.7 default target arm64-apple-darwin23.6.0
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/local/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/Library/Frameworks"
#include "..." search starts here:
#include <...> search starts here:
 /opt/homebrew/Cellar/llvm@15/15.0.7/lib/clang/15.0.7/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include
 /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/System/Library/Frameworks (framework directory)
End of search list.

❯ /Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/tools/clang -v -x c -c - << EOD
    #include <math.h>
EOD
clang version 15.0.7 (/cache/yggdrasil/downloads/clones/llvm-project.git-5a9787eb535c2edc5dea030cc221c1d60f38c9f42344f410e425ea2139e233aa 2593167b92dd2d27849e8bc331db2072a9b4bd7f)
Target: aarch64-apple-darwin23.6.0
Thread model: posix
InstalledDir: /Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/tools
 (in-process)
 "/Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/tools/clang-15" -cc1 -triple arm64-apple-macosx14.0.0 -Wundef-prefix=TARGET_OS_ -Werror=undef-prefix -Wdeprecated-objc-isa-usage -Werror=deprecated-objc-isa-usage -emit-obj -mrelax-all --mrelax-relocations -disable-free -clear-ast-before-backend -disable-llvm-verifier -discard-value-names -main-file-name - -mrelocation-model pic -pic-level 2 -mframe-pointer=non-leaf -ffp-contract=on -fno-rounding-math -funwind-tables=2 -fcompatibility-qualified-id-block-type-checking -fvisibility-inlines-hidden-static-local-var -target-cpu apple-m1 -target-feature +v8.5a -target-feature +crc -target-feature +lse -target-feature +rdm -target-feature +crypto -target-feature +dotprod -target-feature +fp-armv8 -target-feature +neon -target-feature +fp16fml -target-feature +ras -target-feature +rcpc -target-feature +zcm -target-feature +zcz -target-feature +fullfp16 -target-feature +sm4 -target-feature +sha3 -target-feature +sha2 -target-feature +aes -target-abi darwinpcs -fallow-half-arguments-and-returns -mllvm -treat-scalable-fixed-error-as-warning -debugger-tuning=lldb -target-linker-version 530 -v -fcoverage-compilation-dir=/Users/tim/Julia/src/pocl/build/lib/kernel/host -resource-dir /Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/lib/clang/15.0.7 -internal-isystem /usr/local/include -internal-isystem /Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/lib/clang/15.0.7/include -internal-externc-isystem /usr/include -fdebug-compilation-dir=/Users/tim/Julia/src/pocl/build/lib/kernel/host -ferror-limit 19 -stack-protector 1 -fblocks -fencode-extended-block-signature -fregister-global-dtors-with-atexit -fgnuc-version=4.2.1 -fmax-type-align=16 -fcolor-diagnostics -D__GCC_HAVE_DWARF2_CFI_ASM=1 -o -.o -x c -
clang -cc1 version 15.0.7 based upon LLVM 15.0.7jl default target aarch64-apple-darwin23.6.0
ignoring nonexistent directory "/usr/local/include"
ignoring nonexistent directory "/usr/include"
#include "..." search starts here:
#include <...> search starts here:
 /Users/tim/Julia/depot/artifacts/bc0ef6f5449e8f49ae78ed6072c3cf51ea2c9dda/lib/clang/15.0.7/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.
<stdin>:1:14: fatal error: 'math.h' file not found
    #include <math.h>
             ^~~~~~~~
1 error generated.

math.h is located in /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk/usr/include, which is only discovered by the system Clang.

Looks like this is related to sysroot discovery; adding -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk or defining SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk "fixes" the issue.

[maleadt]

Looks like this is hard-coded in the build by Homebrew through -DDEFAULT_SYSROOT: https://github.com/Homebrew/homebrew-core/blob/753cb488e0c6ad28b1f123a8c2a3b9c0337193bd/Formula/l/llvm%4015.rb#L134

... where macos_sdk comes from MacOS.sdk_path_if_needed:

❯ brew ruby -e 'puts MacOS.sdk_path_if_needed'
/Library/Developer/CommandLineTools/SDKs/MacOSX14.sdk

[giordano]

Yeah, you need to pass the path to the SDK with something like

-syslibroot $(xcrun --sdk macosx --show-sdk-path)

see the thread at JuliaLang/julia#47184 (comment)

[maleadt]

Right, I guess I'm arguing that we should configure this by default by setting DEFAULT_SYSROOT. Otherwise Clang_jll is pretty useless without this additional flag, which may not always be easy to set (e.g., POCL_jll is also broken because of this: pocl/pocl#1530). Or is that not possible?

[giordano]

I'm not sure ww can set a default sysroot path, because it depends on the macos version. I guess in homebrew they can do it because their packages are macos version-specific.

[vchuravy]

The primary user of Clang_jll is Clang.jl which uses https://github.com/JuliaInterop/Clang.jl/blob/5e75b5f1b5321c33e6b5ece9cdf05b7e5b325f24/src/generator/context.jl#L243-L245 and https://github.com/JuliaInterop/Clang.jl/blob/5e75b5f1b5321c33e6b5ece9cdf05b7e5b325f24/src/platform/system.jl#L56 but interestingly no isysroot.

In ClangCompiler.jl we did end up setting it https://github.com/Gnimuc/ClangCompiler.jl/blob/acde0348968aae4afad4513fe08c22f611db0d59/examples/clang-interpreter/interpreter.jl#L32

[maleadt]

I'm not sure ww can set a default sysroot path, because it depends on the macos version.

There is a symlink:

❯ ls -lah /Library/Developer/CommandLineTools/SDKs
total 0
drwxr-xr-x  7 root  wheel   224B Mar  8 14:18 .
drwxr-xr-x  5 root  wheel   160B Dec 11  2023 ..
lrwxr-xr-x  1 root  wheel    14B Mar  8 14:17 MacOSX.sdk -> MacOSX14.4.sdk
drwxr-xr-x  7 root  wheel   224B Mar  8 14:18 MacOSX13.3.sdk
lrwxr-xr-x  1 root  wheel    14B Mar  8 14:17 MacOSX13.sdk -> MacOSX13.3.sdk
drwxr-xr-x  7 root  wheel   224B Feb 20 01:28 MacOSX14.4.sdk
lrwxr-xr-x  1 root  wheel    14B Mar  8 14:16 MacOSX14.sdk -> MacOSX14.4.sdk

[vchuravy]

This raises a question I always had. Could we do "CI" for our jlls? Right now we are doing verification, but I have always been hesitating to recommend using clang from Clang_jll since we do no work to verify that it is functional for C/C++.

[maleadt]

I could just paper over this by setting SDKROOT=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk in pocl_jll.jl's init function, but that feels icky.

[vchuravy]

Yeah this reminded me of JuliaPackaging/BinaryBuilder.jl#687

I often have something like: https://github.com/JuliaParallel/PMIx.jl/blob/b5c6be4fe342bae7fc6ee770e1e5cc9d388ceab1/src/api.jl#L11

[vchuravy]

We do know how have JuliaPackaging/BinaryBuilder.jl#791 but that is a similar level of icky