Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rust bindings fail to build built-in backends on Darwin aarch64 #1694

Open
eliasboegel opened this issue Oct 15, 2024 · 13 comments
Open

Rust bindings fail to build built-in backends on Darwin aarch64 #1694

eliasboegel opened this issue Oct 15, 2024 · 13 comments

Comments

@eliasboegel
Copy link

eliasboegel commented Oct 15, 2024

Hi,

I was just trying to build both my own application as well as the Rust examples on a new machine, running an aarch64 Darwin system. The build of the interface appears to succeed (build process shows no errors), however running with any backend (even /cpu/self/ref/* and /cpu/self/opt/*) will result in Backend not currently compiled: /cpu/self Consult the installation instructions to compile this backend.

I am currently using a standard Clang 16.0.6 toolchain, however the same issue occurs with an Apple Clang (15) distribution. Building and running the tests of the C library works without issue, so I don't see this being an issue with the make configuration. I'm currently still unsure if this is a libCEED issue with Bindgen or purely a Bindgen problem. It is also unclear if this is a Darwin problem, or an aarch64 problem as I don't currently have neither an Intel Darwin or Linux aarch64 machine available.

Have the libCEED Rust bindings previously successfully built under Darwin, especially on an aarch64 machine, or are there any ideas what the issue could be? My (unsubstantiated) suspicion is a linker issue.

@jedbrown
Copy link
Member

Are you using static or shared libraries? Does it work from C when you make the same choice? I suspect it's a linker interaction with weak symbols, but would like to narrow down how.

@eliasboegel
Copy link
Author

eliasboegel commented Oct 15, 2024

I was building this in the default configuration of the bindings (i.e. static link). I just tried to build ceedexamples with static linking, and indeed this fails too. Looks like a linking issue like you say. It looks like this happens both with the GCC and with the LLVM linker.

@jedbrown
Copy link
Member

Can you build with make V=1 STATIC=1 and paste the link line? It should look something like this with the two ceed-*-weak.o coming at the end.

ar crD lib/libceed.a build/interface/ceed-basis.o build/interface/ceed.o build/interface/ceed-elemrestriction.o build/interface/ceed-fortran.o build/interface/ceed-jit-source-root-default.o build/interface/ceed-jit-tools.o build/interface/ceed-operator.o build/interface/ceed-preconditioning.o build/interface/ceed-qfunction.o build/interface/ceed-qfunctioncontext.o build/interface/ceed-qfunction-register.o build/interface/ceed-register.o build/interface/ceed-tensor.o build/interface/ceed-types.o build/interface/ceed-vector.o build/gallery/identity/ceed-identity.o build/gallery/mass/ceed-mass1dbuild.o build/gallery/mass/ceed-mass2dbuild.o build/gallery/mass/ceed-mass3dbuild.o build/gallery/mass/ceed-massapply.o build/gallery/mass-vector/ceed-vectormassapply.o build/gallery/poisson/ceed-poisson1dapply.o build/gallery/poisson/ceed-poisson1dbuild.o build/gallery/poisson/ceed-poisson2dapply.o build/gallery/poisson/ceed-poisson2dbuild.o build/gallery/poisson/ceed-poisson3dapply.o build/gallery/poisson/ceed-poisson3dbuild.o build/gallery/poisson-vector/ceed-vectorpoisson1dapply.o build/gallery/poisson-vector/ceed-vectorpoisson2dapply.o build/gallery/poisson-vector/ceed-vectorpoisson3dapply.o build/gallery/scale/ceed-scale.o build/backends/ref/ceed-ref-basis.o build/backends/ref/ceed-ref-operator.o build/backends/ref/ceed-ref-qfunction.o build/backends/ref/ceed-ref-qfunctioncontext.o build/backends/ref/ceed-ref-restriction.o build/backends/ref/ceed-ref-tensor.o build/backends/ref/ceed-ref-vector.o build/backends/ref/ceed-ref.o build/backends/blocked/ceed-blocked-operator.o build/backends/blocked/ceed-blocked.o build/backends/opt/ceed-opt-blocked.o build/backends/opt/ceed-opt-operator.o build/backends/opt/ceed-opt-serial.o build/backends/opt/ceed-opt-tensor.o build/backends/memcheck/ceed-memcheck-blocked.o build/backends/memcheck/ceed-memcheck-qfunction.o build/backends/memcheck/ceed-memcheck-qfunctioncontext.o build/backends/memcheck/ceed-memcheck-restriction.o build/backends/memcheck/ceed-memcheck-serial.o build/backends/memcheck/ceed-memcheck-vector.o build/backends/avx/ceed-avx-blocked.o build/backends/avx/ceed-avx-serial.o build/backends/avx/ceed-avx-tensor.o build/backends/ceed-backend-weak.o build/gallery/ceed-gallery-weak.o

Could you also show output of nm lib/libceed.a | grep CeedRegister_?

@eliasboegel
Copy link
Author

Can you build with make V=1 STATIC=1 and paste the link line? It should look something like this with the two ceed-*-weak.o coming at the end.

ar crD lib/libceed.a build/interface/ceed-basis.o build/interface/ceed.o build/interface/ceed-elemrestriction.o build/interface/ceed-fortran.o build/interface/ceed-jit-source-root-default.o build/interface/ceed-jit-tools.o build/interface/ceed-operator.o build/interface/ceed-preconditioning.o build/interface/ceed-qfunction.o build/interface/ceed-qfunctioncontext.o build/interface/ceed-qfunction-register.o build/interface/ceed-register.o build/interface/ceed-tensor.o build/interface/ceed-types.o build/interface/ceed-vector.o build/gallery/identity/ceed-identity.o build/gallery/mass/ceed-mass1dbuild.o build/gallery/mass/ceed-mass2dbuild.o build/gallery/mass/ceed-mass3dbuild.o build/gallery/mass/ceed-massapply.o build/gallery/mass-vector/ceed-vectormassapply.o build/gallery/poisson/ceed-poisson1dapply.o build/gallery/poisson/ceed-poisson1dbuild.o build/gallery/poisson/ceed-poisson2dapply.o build/gallery/poisson/ceed-poisson2dbuild.o build/gallery/poisson/ceed-poisson3dapply.o build/gallery/poisson/ceed-poisson3dbuild.o build/gallery/poisson-vector/ceed-vectorpoisson1dapply.o build/gallery/poisson-vector/ceed-vectorpoisson2dapply.o build/gallery/poisson-vector/ceed-vectorpoisson3dapply.o build/gallery/scale/ceed-scale.o build/backends/ref/ceed-ref-basis.o build/backends/ref/ceed-ref-operator.o build/backends/ref/ceed-ref-qfunction.o build/backends/ref/ceed-ref-qfunctioncontext.o build/backends/ref/ceed-ref-restriction.o build/backends/ref/ceed-ref-tensor.o build/backends/ref/ceed-ref-vector.o build/backends/ref/ceed-ref.o build/backends/blocked/ceed-blocked-operator.o build/backends/blocked/ceed-blocked.o build/backends/opt/ceed-opt-blocked.o build/backends/opt/ceed-opt-operator.o build/backends/opt/ceed-opt-serial.o build/backends/opt/ceed-opt-tensor.o build/backends/memcheck/ceed-memcheck-blocked.o build/backends/memcheck/ceed-memcheck-qfunction.o build/backends/memcheck/ceed-memcheck-qfunctioncontext.o build/backends/memcheck/ceed-memcheck-restriction.o build/backends/memcheck/ceed-memcheck-serial.o build/backends/memcheck/ceed-memcheck-vector.o build/backends/avx/ceed-avx-blocked.o build/backends/avx/ceed-avx-serial.o build/backends/avx/ceed-avx-tensor.o build/backends/ceed-backend-weak.o build/gallery/ceed-gallery-weak.o
ar crD lib/libceed.a build/interface/ceed-basis.o build/interface/ceed-elemrestriction.o build/interface/ceed-fortran.o build/interface/ceed-jit-source-root-default.o build/interface/ceed-jit-tools.o build/interface/ceed-operator.o build/interface/ceed-preconditioning.o build/interface/ceed-qfunction-register.o build/interface/ceed-qfunction.o build/interface/ceed-qfunctioncontext.o build/interface/ceed-register.o build/interface/ceed-tensor.o build/interface/ceed-types.o build/interface/ceed-vector.o build/interface/ceed.o build/gallery/identity/ceed-identity.o build/gallery/mass-vector/ceed-vectormassapply.o build/gallery/mass/ceed-mass1dbuild.o build/gallery/mass/ceed-mass2dbuild.o build/gallery/mass/ceed-mass3dbuild.o build/gallery/mass/ceed-massapply.o build/gallery/poisson-vector/ceed-vectorpoisson1dapply.o build/gallery/poisson-vector/ceed-vectorpoisson2dapply.o build/gallery/poisson-vector/ceed-vectorpoisson3dapply.o build/gallery/poisson/ceed-poisson1dapply.o build/gallery/poisson/ceed-poisson1dbuild.o build/gallery/poisson/ceed-poisson2dapply.o build/gallery/poisson/ceed-poisson2dbuild.o build/gallery/poisson/ceed-poisson3dapply.o build/gallery/poisson/ceed-poisson3dbuild.o build/gallery/scale/ceed-scale.o build/backends/ref/ceed-ref-basis.o build/backends/ref/ceed-ref-operator.o build/backends/ref/ceed-ref-qfunction.o build/backends/ref/ceed-ref-qfunctioncontext.o build/backends/ref/ceed-ref-restriction.o build/backends/ref/ceed-ref-tensor.o build/backends/ref/ceed-ref-vector.o build/backends/ref/ceed-ref.o build/backends/blocked/ceed-blocked-operator.o build/backends/blocked/ceed-blocked.o build/backends/opt/ceed-opt-blocked.o build/backends/opt/ceed-opt-operator.o build/backends/opt/ceed-opt-serial.o build/backends/opt/ceed-opt-tensor.o build/backends/ceed-backend-weak.o build/gallery/ceed-gallery-weak.o

Could you also show output of nm lib/libceed.a | grep CeedRegister_?

                 U _CeedRegister_Avx_Blocked
                 U _CeedRegister_Avx_Serial
                 U _CeedRegister_Cuda
                 U _CeedRegister_Cuda_Gen
                 U _CeedRegister_Cuda_Shared
                 U _CeedRegister_Hip
                 U _CeedRegister_Hip_Gen
                 U _CeedRegister_Hip_Shared
                 U _CeedRegister_Magma
                 U _CeedRegister_Magma_Det
                 U _CeedRegister_Memcheck_Blocked
                 U _CeedRegister_Memcheck_Serial
                 U _CeedRegister_Occa
                 U _CeedRegister_Opt_Blocked
                 U _CeedRegister_Opt_Serial
                 U _CeedRegister_Ref
                 U _CeedRegister_Ref_Blocked
                 U _CeedRegister_Sycl
                 U _CeedRegister_Sycl_Gen
                 U _CeedRegister_Sycl_Shared
                 U _CeedRegister_Xsmm_Blocked
                 U _CeedRegister_Xsmm_Serial
0000000000000000 T _CeedRegister_Ref
0000000000000000 T _CeedRegister_Ref_Blocked
0000000000000000 T _CeedRegister_Opt_Blocked
0000000000000000 T _CeedRegister_Opt_Serial
0000000000000000 T _CeedRegister_Avx_Blocked
000000000000013c T _CeedRegister_Avx_Serial
0000000000000170 T _CeedRegister_Cuda
00000000000001a4 T _CeedRegister_Cuda_Gen
00000000000001d8 T _CeedRegister_Cuda_Shared
000000000000020c T _CeedRegister_Hip
0000000000000240 T _CeedRegister_Hip_Gen
0000000000000274 T _CeedRegister_Hip_Shared
0000000000000344 T _CeedRegister_Magma
0000000000000384 T _CeedRegister_Magma_Det
00000000000003c4 T _CeedRegister_Memcheck_Blocked
00000000000003f8 T _CeedRegister_Memcheck_Serial
000000000000042c T _CeedRegister_Occa
0000000000000498 T _CeedRegister_Opt_Blocked
00000000000004cc T _CeedRegister_Opt_Serial
0000000000000500 T _CeedRegister_Ref
0000000000000534 T _CeedRegister_Ref_Blocked
00000000000002a8 T _CeedRegister_Sycl
0000000000000310 T _CeedRegister_Sycl_Gen
00000000000002dc T _CeedRegister_Sycl_Shared
0000000000000034 t _CeedRegister_Weak
0000000000000568 T _CeedRegister_Xsmm_Blocked
000000000000059c T _CeedRegister_Xsmm_Serial

@jedbrown
Copy link
Member

Hmm, seems the weak attribute is being lost. Do you know how to make those by a W as they should be?

$ nm lib/libceed.a |rg CeedRegister_
                 U CeedRegister_Avx_Blocked
                 U CeedRegister_Avx_Serial
                 U CeedRegister_Cuda
                 U CeedRegister_Cuda_Gen
                 U CeedRegister_Cuda_Shared
                 U CeedRegister_Hip
                 U CeedRegister_Hip_Gen
                 U CeedRegister_Hip_Shared
                 U CeedRegister_Magma
                 U CeedRegister_Magma_Det
                 U CeedRegister_Memcheck_Blocked
                 U CeedRegister_Memcheck_Serial
                 U CeedRegister_Occa
                 U CeedRegister_Opt_Blocked
                 U CeedRegister_Opt_Serial
                 U CeedRegister_Ref
                 U CeedRegister_Ref_Blocked
                 U CeedRegister_Sycl
                 U CeedRegister_Sycl_Gen
                 U CeedRegister_Sycl_Shared
                 U CeedRegister_Xsmm_Blocked
                 U CeedRegister_Xsmm_Serial
000000000000032c T CeedRegister_Ref
0000000000000124 T CeedRegister_Ref_Blocked
0000000000000242 T CeedRegister_Opt_Blocked
000000000000022f T CeedRegister_Opt_Serial
00000000000001e8 T CeedRegister_Memcheck_Blocked
00000000000001fb T CeedRegister_Memcheck_Serial
0000000000000137 T CeedRegister_Avx_Blocked
0000000000000124 T CeedRegister_Avx_Serial
0000000000000132 W CeedRegister_Avx_Blocked
0000000000000158 W CeedRegister_Avx_Serial
000000000000017e W CeedRegister_Cuda
00000000000001a4 W CeedRegister_Cuda_Gen
00000000000001ca W CeedRegister_Cuda_Shared
00000000000001f0 W CeedRegister_Hip
0000000000000216 W CeedRegister_Hip_Gen
000000000000023c W CeedRegister_Hip_Shared
00000000000002d4 W CeedRegister_Magma
0000000000000301 W CeedRegister_Magma_Det
000000000000032e W CeedRegister_Memcheck_Blocked
0000000000000354 W CeedRegister_Memcheck_Serial
000000000000037a W CeedRegister_Occa
00000000000003c5 W CeedRegister_Opt_Blocked
00000000000003eb W CeedRegister_Opt_Serial
0000000000000411 W CeedRegister_Ref
0000000000000437 W CeedRegister_Ref_Blocked
0000000000000262 W CeedRegister_Sycl
00000000000002ae W CeedRegister_Sycl_Gen
0000000000000288 W CeedRegister_Sycl_Shared
0000000000000000 t CeedRegister_Weak
000000000000045d W CeedRegister_Xsmm_Blocked
0000000000000483 W CeedRegister_Xsmm_Serial

What we do now is for backends/ceed-backend-weak.c to define a symbol with __attribute__((weak)) that should be used only if the symbol has not yet been defined. In the above, you see the T symbols have been defined (in various source backends/* source files, the object files of which should come first on the link line) and weak (W) symbols are picked up only for the remaining symbols. It appears that MacOS (or the linker being used here) is discarding the regular symbols and promoting weak symbols to T.

@eliasboegel
Copy link
Author

eliasboegel commented Oct 16, 2024

Unfortunately I have no idea so far. I've now tried a range of different things from different toolchains (GCC 12, LLVM, Apple LLVM) to both different weak symbol markers (Apple "prefers" __attribute__((weak_import))) and a couple additional flags. I am now quite certain that the issue is not the linker but the way clang/gcc work on MacOS. I came to this conclusion because ceed-backend-weak.o already does not have its symbols marked as weak, and this is pre-ar and pre-ld. I would imagine this is a widespread problem on MacOS, but there is surprisingly little to be found about the issue. Nevertheless, thanks for the help in narrowing it down.

@eliasboegel
Copy link
Author

eliasboegel commented Oct 16, 2024

What I've done now as a workaround is the following:
I modified the Rust bindings such that libceed-sys features are re-exported/replicated in libceed such that libCEED can be build as a dynamic library by disabling the libceed-sys static feature like this:

libceed = { version/git, default-features = false }

This works and, while ugly, this allows building libCEED as a dynamic library only on MacOS:

[target.'cfg(not(target_os = "macos"))'.dependencies]
libceed = { version/git }
[target.'cfg(target_os = "macos")'.dependencies]
libceed = { version/git, default-features = false }

This "fixes" my problem for now. Are you interested in having the small change in libceed Cargo.toml to re-export libceed-sys features upstreamed?

@jedbrown
Copy link
Member

Does that end up requiring DYLD_LIBRARY_PATH? Ideally the features (static and system) would really be orthogonal, though I see that's not the case currently. Are you having cargo build a libceed.dylib or have you built that in advance?

@eliasboegel
Copy link
Author

eliasboegel commented Oct 16, 2024

I have cargo build libceed.dylib. I did not have to deal with DYLD_LIBRARY_PATH, but I've only tested on my machine since I don't have access to another MacOS device.

Also, I do not use system in my workaround, I just figured I would make this feature accessible through libceed too. Correct me if I'm wrong, but my understanding is that there is no way to set features of the libceed-sys subdependency of a project using only libceed.

@jedbrown
Copy link
Member

I'm concerned there may be another libceed.dylib in your search path, but maybe that has been handled automatically for MacOS. This is a way to automate that on Linux. https://gitlab.com/petsc/petsc-rs/-/blob/main/examples/snes/build.rs?ref_type=heads

Can we put the default feature selection entirely inside libceed-sys?

@eliasboegel
Copy link
Author

I'm concerned there may be another libceed.dylib in your search path, but maybe that has been handled automatically for MacOS. This is a way to automate that on Linux. https://gitlab.com/petsc/petsc-rs/-/blob/main/examples/snes/build.rs?ref_type=heads

I just ran find / -name libceed.dylib and the only hits were those generated by libceed-sys:

project_dir/target/debug/build/libceed-sys-139534ee89f6dc46/out/lib/libceed.dylib
project_dir/target/debug/build/libceed-sys-139534ee89f6dc46/out/build/lib/libceed.dylib

They were generated with cargo clean followed by cargo build in an environment with reset environment variables. If anything, Cargo on MacOS handles the path of the generated libceed.dylib automatically.

Can we put the default feature selection entirely inside libceed-sys?

My understanding is that this used to be possible with the feature resolver V1 of Cargo, provided features are only added instead of removed with default-features = false because of feature unification. That would have made it possible to rework libceed-sys features to use a non-default feature to build a shared/dynamic library and also adding libceed-sys to the application's Cargo.toml with this feature. Now that feature resolver V2 was introduced for Rust editions >= 2021, I don't think this is possible any longer, so re-exporting the libceed-sys features in libceed seems necessary for users of libceed(as opposed to direct use of libceed-sys).

@jedbrown
Copy link
Member

Okay, thanks. Would you mind putting what you're thinking in a branch? I still wonder if there is some mixing of Apple and gcc/clang upstream linking tools that is interfering with weak symbols, but perhaps it just doesn't work with static libs?

@eliasboegel
Copy link
Author

eliasboegel commented Oct 17, 2024

I currently keep my small change in https://github.com/eliasboegel/libCEED. I use this as the Cargo dependency in my project, and disable static with default-features = false on MacOS. I'm happy to put this into a branch in the main repository if you need me to.

At this point of static linking I also want to add that I did try a minimum example of weak linking on my MacOS machine - just a simple function in a static library as a weak symbol. I found that, while the symbols still weren't marked as W, the linker had no problem overriding my weak marked function with a strong one when I actually tried it. Perhaps MacOS has a different way in which symbols are marked weak that nm does not show?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants