Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: note musl based static builds, that work across linux distros + alpine ffi static & shared libs #416

Closed
YOU54F opened this issue May 10, 2024 · 5 comments

Comments

@YOU54F
Copy link
Member

YOU54F commented May 10, 2024

it would be good to document this on the readme, so users are aware that

  • Alpine is now supported
  • The builds are static, and therefore the single binary works across musl or glibc based distros (excluding ffi)
  • pact_ffi library comes in both static *.a & shared *.so variants for musl, depending on the consuming client libraries requirements for ffi interop
library alpine support version
pact_ffi - x86_64 .a from 0.2.4
- aarch64 .a from 0.4.15
- .so from 0.4.17
pact_mock_server_cli - 1.0.5
pact_verifier_cli - 1.1.1
pact-stub-server - 0.6.0
pact-plugin-cli - 0.1.2
pact-protobuf-plugin - 0.3.15
pact-csv-plugin - 0.0.6
@mefellows
Copy link
Member

Nice! We should update here also: https://docs.pact.io/docker

@YOU54F
Copy link
Member Author

YOU54F commented May 14, 2024

So we have an issue with the pact_ffi shared library, for aarch64-musl .so in 0.4.20

Error

LoadError:
  Could not open library '/app/lib/pact/../../ffi/linux-arm64-musl/libpact_ffi.so': Error relocating /app/lib/pact/../../ffi/linux-arm64-musl/libpact_ffi.so: __builtin_copysignq: symbol not found.
  Searched in <system library path>, /usr/lib, /usr/local/lib, /opt/local/lib
# /usr/local/bundle/gems/ffi-1.16.3/lib/ffi/dynamic_library.rb:65:in `load_library'

Seems to be exclusive to aarch64, and the dynamic library only. amd64 tests are passing in pact-php as well using the shared library, and the static .a is being used and tested successfully in pact-js-core in this PR

First reported by @tienvx over in pact-php alpine PR - Thanks Tien!

@YOU54F
Copy link
Member Author

YOU54F commented May 15, 2024

So I've just built the aarch64 musl libs locally on my macbook,

build_aarch64_musl() {
install_cross
cargo clean
cross build --target aarch64-unknown-linux-musl "${cargo_flags[@]}"
if [[ "${cargo_flags[*]}" =~ "--release" ]]; then
BUILD_SCRIPT=$(cat <<EOM
apk add --no-cache musl-dev gcc && \
cd /scratch && \
ar -x libpact_ffi.a && \
gcc -shared *.o -o libpact_ffi.so && \
rm -f *.o
EOM
)
docker run \
--platform=linux/arm64 \
--rm \
-v "$CARGO_TARGET_DIR/aarch64-unknown-linux-musl/release:/scratch" \
alpine \
/bin/sh -c "$BUILD_SCRIPT"
gzip_and_sum \
"$CARGO_TARGET_DIR/aarch64-unknown-linux-musl/release/libpact_ffi.a" \
"$ARTIFACTS_DIR/libpact_ffi-linux-aarch64-musl.a.gz"
gzip_and_sum \
"$CARGO_TARGET_DIR/aarch64-unknown-linux-musl/release/libpact_ffi.so" \
"$ARTIFACTS_DIR/libpact_ffi-linux-aarch64-musl.so.gz"
fi
}

Copied these over to pact-php

cp ../release_artifacts/libpact_ffi-linux-aarch64-musl.so ~/dev/pact-foundation/pact-php/bin/pact-ffi-lib/pact.so

Green php tests.

Screenshot 2024-05-15 at 18 44 43

note have tried regenerating the .so from the released 0.4.20 .a but get the same missing symbol issue, so not sure what has caused it

YOU54F added a commit to YOU54F/pact-net that referenced this issue May 29, 2024
## Rationale

pact-reference has introduced musl and arm64 based ffi libraries for linux

- pact-foundation/pact-reference#416

Tracking Issue

- pact-foundation/roadmap#30

## Issues Resolved

fixes pact-foundation#498
fixes pact-foundation#496
fixes pact-foundation#500
fixes pact-foundation#374
fixes pact-foundation#387

## Backwards Compatibility

Linux glibc based hosts take precedence, so if any error occurs during musl
detection. I do not anticipate breaking changes for users

## Implementation notes

### .NET notes

- Docs
  - [Uses MSBuild Exec task](https://learn.microsoft.com/en-us/visualstudio/msbuild/exec-task?view=vs-2022)
- MSBuild Blog Posts
  - [Cross-Platform Build Events in .NET Core using MSBuild](https://jeremybytes.blogspot.com/2020/05/cross-platform-build-events-in-net-core.html)
  - [MSBuild 101: Using the exit code from a command](https://www.creepingcoder.com/2020/06/01/msbuild-101-using-the-exit-code-from-a-command/)
- Stack OverFlow
  - [Set PropertyGroup property to Exec output](https://stackoverflow.com/questions/76583824/set-propertygroup-property-to-exec-output)
- .NET runtime musl detection code
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/mono/llvm/llvm-init.proj\#L7
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.Unix.cs\#L78t

### Conditions for execution

musl detection will run if

- if linux
- if /lib/ld-musl-(x86_64|aarch64).so.1 exists
- if ldd bin/sh | grep musl is true (musl lib is loaded, rather than glibc)

will continue on error, reverting back to glibc based libaries.

### Supported musl targets

should work for multiple musl based distroes if

- /lib/ld-musl-(x86_64|aarch64).so.1 exists
- ldd is available (available by default in alpine images)

Tested on Alpine ARM64 / AMD64.

## Caveats

- [.NET does not run under QEMU](https://github.com/dotnet/core/blob/main/release-notes/8.0/supported-os.md#qemu) affecting the ability to test multi-arch from a single system
- .NET restore can take a long time when running under containers.
  - [Workaround](NuGet/Home#13062 (comment)): Set `DOTNET_NUGET_SIGNATURE_VERIFICATION` to `false`

## Compatibility

### Operating System

Due to using a shared native library instead of C# for the main Pact logic only certain OSs are supported:

| OS           | Arch        | Support                                                            |
| ------------ | ----------- | -------------------------------------------------------------------|
| Windows      | x86         | ❌ No                                                              |
| Windows      | x64         | ✔️ Yes                                                              |
| Linux (libc) | x86         | ❌ No                                                              |
| Linux (libc) | x64         | ✔️ Yes                                                              |
| Linux (musl) | x64         | ✔️ Yes (Tier 2)*                                                    |
| Linux (libc) | ARM         | ✔️ Yes (Tier 3)*                                                    |
| Linux (musl) | ARM         | ✔️ Yes (Tier 3)*                                                    |
| OSX          | x64         | ✔️ Yes                                                              |
| OSX          | ARM (M1/M2) | ✔️ Yes                                                              |

#### Support

- Tier 1
  - Established
  - Full CI/CD support.
  - Users should not encounter issues
  - Full reproducible examples running in CI, should be provided by users raising issues
  - If using musl targets, users should attempt the same test on a libc target (such as debian)
- Tier 2
  - Recently introduced
  - Full CI/CD support.
  - Users may encounter issues
  - Full reproducible examples running in CI, should be provided by users raising issues
  - If using musl targets, users should attempt the same test on a libc target (such as debian)
- Tier 3
  - Recently introduced, No/limited CI/CD support.
  - Users may encounter issues
  - Full reproducible examples which can be run by maintainers locally, should be provided by users raising issues
@YOU54F
Copy link
Member Author

YOU54F commented Jun 5, 2024

I think this was due to stripping of the symbols from the static archive, prior to generating the shared object file. It's possible to force the generation of the *.so at compile time with some rust flags, so that has been performed along with a smoke test to ensure its loading correctly in #432

@YOU54F YOU54F closed this as completed in fe07b87 Sep 6, 2024
@YOU54F
Copy link
Member Author

YOU54F commented Sep 6, 2024

Updated compat docs now added to pact_ffi readme

https://github.com/pact-foundation/pact-reference/tree/master/rust/pact_ffi#compatibility

Nice! We should update here also: https://docs.pact.io/docker

Updated PR for docs page here

pact-foundation/docs.pact.io#345

YOU54F added a commit to YOU54F/pact-net that referenced this issue Sep 24, 2024
## Rationale

pact-reference has introduced musl and arm64 based ffi libraries for linux

- pact-foundation/pact-reference#416

Tracking Issue

- pact-foundation/roadmap#30

## Issues Resolved

fixes pact-foundation#498
fixes pact-foundation#496
fixes pact-foundation#500
fixes pact-foundation#374
fixes pact-foundation#387

## Backwards Compatibility

Linux glibc based hosts take precedence, so if any error occurs during musl
detection. I do not anticipate breaking changes for users

## Implementation notes

### .NET notes

- Docs
  - [Uses MSBuild Exec task](https://learn.microsoft.com/en-us/visualstudio/msbuild/exec-task?view=vs-2022)
- MSBuild Blog Posts
  - [Cross-Platform Build Events in .NET Core using MSBuild](https://jeremybytes.blogspot.com/2020/05/cross-platform-build-events-in-net-core.html)
  - [MSBuild 101: Using the exit code from a command](https://www.creepingcoder.com/2020/06/01/msbuild-101-using-the-exit-code-from-a-command/)
- Stack OverFlow
  - [Set PropertyGroup property to Exec output](https://stackoverflow.com/questions/76583824/set-propertygroup-property-to-exec-output)
- .NET runtime musl detection code
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/mono/llvm/llvm-init.proj\#L7
  - https://github.com/dotnet/runtime/blob/a50ba0669353893ca8ade8568b0a7d210b5a425f/src/libraries/Common/tests/TestUtilities/System/PlatformDetection.Unix.cs\#L78t

### Conditions for execution

musl detection will run if

- if linux
- if /lib/ld-musl-(x86_64|aarch64).so.1 exists
- if ldd bin/sh | grep musl is true (musl lib is loaded, rather than glibc)

will continue on error, reverting back to glibc based libaries.

### Supported musl targets

should work for multiple musl based distroes if

- /lib/ld-musl-(x86_64|aarch64).so.1 exists
- ldd is available (available by default in alpine images)

Tested on Alpine ARM64 / AMD64.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants