Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem about LIBBPF_SYS_LIBRARY_PATH and build.rs #106

Open
kxxt opened this issue Aug 29, 2024 · 8 comments
Open

Problem about LIBBPF_SYS_LIBRARY_PATH and build.rs #106

kxxt opened this issue Aug 29, 2024 · 8 comments

Comments

@kxxt
Copy link
Contributor

kxxt commented Aug 29, 2024

When cross-compiling an application that uses libbpf-sys in build.rs(Usually via libbpf-cargo).

If LIBBPF_SYS_LIBRARY_PATH is set to a path that contains the libelf for the target architecture(let's say, aarch64),

then while building the build.rs of this application, we are actually required to build libbpf-sys for the build architecture(x86_64) because the build.rs script needs to link libbpf-sys. But in the build.rs of libbpf-sys, we can only know that we are building for aarch64 target and we are linking aarch64 libelf from LIBBPF_SYS_LIBRARY_PATH into x86_64 libbpf-sys rlib.

static:

 = note: rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_version.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_hash.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_error.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_fill.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_begin.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_next.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_rand.o) is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/target/debug/deps/liblibbpf_sys-abc8218b23be072d.rlib(elf_end.o) is incompatible with elf64-x86-64

dynamic:

  = note: rust-lld: error: /home/kxxt/repos/tracexec/3rdparty/aarch64/usr/lib/aarch64-linux-gnu/libelf.so is incompatible with elf64-x86-64
          rust-lld: error: /home/kxxt/repos/tracexec/3rdparty/aarch64/usr/lib/aarch64-linux-gnu/libz.so is incompatible with elf64-x86-64
          collect2: error: ld returned 1 exit status

This almost makes LIBBPF_SYS_LIBRARY_PATH useless unless libbpf-sys is only used as a dependency but not a build dependency at the same time.

@danielocfb
Copy link
Collaborator

Would be best if you could share a sample project and build invocation. Perhaps we have to introduce a variant of this variable to be used in a cross compilation context?

@kxxt
Copy link
Contributor Author

kxxt commented Aug 30, 2024

Would be best if you could share a sample project and build invocation.

I have made a minimal reproduction repo based on the capable example: https://github.com/kxxt/libbpf-sys-106-repro

Running the build-aarch64.sh script should reproduce this bug.

Perhaps we have to introduce a variant of this variable to be used in a cross compilation context?

But the problem is how could we determine the cross compilation context?

@danielocfb
Copy link
Collaborator

danielocfb commented Sep 3, 2024

I have made a minimal reproduction repo based on the capable example: https://github.com/kxxt/libbpf-sys-106-repro

Okay, thanks for this, that's tremendously useful. I finally got the error reproduced.

But the problem is how could we determine the cross compilation context?

I'd think we will be cross-compiling if CARGO_CFG_TARGET_ARCH != std::env::consts::ARCH.

@danielocfb
Copy link
Collaborator

Yeah, so if I adjust your script as follows:

--- build-aarch64.sh
+++ build-aarch64.sh
@@ -1,6 +1,8 @@
 #!/bin/sh

-export LIBBPF_SYS_LIBRARY_PATH="$(realpath 3rdparty/aarch64/usr/lib/aarch64-linux-gnu)"
-export LIBBPF_SYS_EXTRA_CFLAGS="-I $(realpath 3rdparty/aarch64/usr/include)"
+export LIBBPF_SYS_LIBRARY_PATH_x86_64="$(realpath 3rdparty/x86_64/usr/lib/x86_64-linux-gnu)"
+export LIBBPF_SYS_EXTRA_CFLAGS_x86_64="-I $(realpath 3rdparty/x86_64/usr/include)"
+export LIBBPF_SYS_LIBRARY_PATH_aarch64="$(realpath 3rdparty/aarch64/usr/lib/aarch64-linux-gnu)"
+export LIBBPF_SYS_EXTRA_CFLAGS_aarch64="-I $(realpath 3rdparty/aarch64/usr/include)"

 cargo build --target aarch64-unknown-linux-gnu -F static

and the libbpf-sys build script like this:

--- build.rs
+++ build.rs
@@ -111,6 +111,7 @@ fn pkg_check(pkg: &str) {

 fn main() {
     let src_dir = path::PathBuf::from(env::var_os("CARGO_MANIFEST_DIR").unwrap());
+    let arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap();

     generate_bindings(src_dir.clone());

@@ -156,8 +157,9 @@ fn main() {
             "a C compiler is required to compile libbpf-sys using the vendored copy of libbpf",
         );
         let mut cflags = compiler.cflags_env();
-        println!("cargo:rerun-if-env-changed=LIBBPF_SYS_EXTRA_CFLAGS");
-        if let Some(extra_cflags) = env::var_os("LIBBPF_SYS_EXTRA_CFLAGS") {
+        let sys_cflags_var = format!("LIBBPF_SYS_EXTRA_CFLAGS_{arch}");
+        println!("cargo:rerun-if-env-changed={sys_cflags_var}");
+        if let Some(extra_cflags) = env::var_os(sys_cflags_var) {
             cflags.push(" ");
             cflags.push(extra_cflags);
         }
@@ -198,8 +200,9 @@ fn main() {
     );
     println!("cargo:include={}/include", out_dir.to_string_lossy());

-    println!("cargo:rerun-if-env-changed=LIBBPF_SYS_LIBRARY_PATH");
-    if let Ok(lib_path) = env::var("LIBBPF_SYS_LIBRARY_PATH") {
+    let sys_path_var = format!("LIBBPF_SYS_LIBRARY_PATH_{arch}");
+    println!("cargo:rerun-if-env-changed={sys_path_var}");
+    if let Ok(lib_path) = env::var(sys_path_var) {
         for path in lib_path.split(':') {
             if !path.is_empty() {
                 println!("cargo:rustc-link-search=native={}", path);

Then I get seemingly further:

  = note: rust-lld: error: undefined symbol: ZSTD_createCCtx
          >>> referenced by elf_compress.o:(__libelf_compress) in archive /tmp/libbpf-sys-106-repro/target/debug/deps/liblibbpf_sys-184db90333a73bf9.rlib

The error seems to be caused by the libraries you vendored in the example project being linked against some elfutils built with zstd support, would be my reading of it. So it's not really relevant here, I think, though it does mask any potentially successful build.

@danielocfb
Copy link
Collaborator

Likely the next problem we hit will be that, in the above scenario, the skeleton will be created for x86, it seems. Which makes sense, because the build script is always run on the host. But I am not sure it's what you'd want. I suspect we'd need to adjust vmlinux path or something to make that work as well (though I don't know whether that will be enough).

@kxxt
Copy link
Contributor Author

kxxt commented Sep 5, 2024

Likely the next problem we hit will be that, in the above scenario, the skeleton will be created for x86, it seems. Which makes sense, because the build script is always run on the host. But I am not sure it's what you'd want. I suspect we'd need to adjust vmlinux path or something to make that work as well (though I don't know whether that will be enough).

I don't think that is a problem. The build script runs on the host, but cargo passes correct CARGO_CFG_TARGET_ARCH to it so the correct header is used.

@danielocfb
Copy link
Collaborator

Have you gotten it to work end-to-end, @kxxt ? If so, feel free to open a pull request.

@kxxt
Copy link
Contributor Author

kxxt commented Sep 7, 2024

Have you gotten it to work end-to-end, @kxxt ? If so, feel free to open a pull request.

No. I didn't try to get it to work. I initially tried to use this variable but find it not working, then switched to what the libbpf-rs CI is doing. But I think it is worth reporting that this variable doesn't work as intended for projects that uses libbpf-cargo or depending on libbpf-sys in any way in build.rs. It's the beginning of a new semester now and I am too busy to work on a solution that I don't actually use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants