chore: Build for x86_64-unknown-linux-musl with all features and optimized binary size

[ghost]

This PR enables all features, including those with native dependencies, for x86_64-unknown-linux-musl target. The resulting binary is fully statically linked, LTO-optimized, and stripped.

The build toolchain is based on Clang and libc++ from LLVM project, in addition to musl.

libc++ is used by rdkafka and leveldb. Although it should be possible to link GNU libstdc++ statically, it makes the resulting binaries licensed under LGPL, which might be not desirable as Vector's license is Apache 2. In contrast, libc++ is MIT/UIUC licensed.

Clang is used not only because it is easy to pair with libc++, but also because it enables cross-language linker plugin LTO. Linker plugin LTO is different from LTO codegen option in Rust. Both types of LTO can be used together to reduce binary size and improve runtime performance.

Here are the measurements of the binary size with different LTO combinations for x86_64-unknown-linux-musl target:

Rust codegen LTO	Linker plugin LTO	Original binary size, MB	Stripped binary size, MB	Compressed stripped binary size, MB
❌	❌	26.78	17.34	6.59
❌	lto=thin	27.34	18.17	6.54
❌	lto	26.98	18.33	6.58
lto=thin	❌	25.76	17.58	6.67
lto=thin	lto=thin	27.35	18.18	6.54
lto	❌	20.95	14.70	5.96
lto	lto	20.11	14.85	5.80

From the results listed above it can be seen that Rust codegen LTO type lto and linker plugin LTO type lto deliver the best size of compressed stripped binary. It theoretically should also have the best performance, as LTO reduces functions call overhead by inlining small functions.

For comparison, a similar table for x86_64-unknown-linux-gnu:

Rust codegen LTO	Linker plugin LTO	Original binary size, MB	Stripped binary size, MB	Compressed stripped binary size, MB
❌	❌	34.50	17.10	6.20
lto=thin	❌	33.40	17.29	6.27
lto	❌	28.64	14.37	5.55

As a result, Rust codegen LTO type lto and linker plugin LTO type lto are used for x86_64-unknown-linux-musl. Rust LTO type is not set in Cargo.toml, but is added at the build time because LTO significantly increases build times, and in my opinion it is better to start from enabling it only for certain build types in CI.

Stripping of Vector binary is added for all target platforms because it can't hurt, but only reduce resulting binaries size.

Ref #661, #630, #654, #669

[binarylogic]

@a-rodin this is fantastic! Thank you for doing this. I spent a considerable amount of time trying to do exactly this and was unsuccessful. In fact, I've been reaching out to our network to try and find someone to help with this, so I'm really happy to see this done. 😄

I'll have @LucioFranco review and then we can merge and get it fully integrated into our releases.

[LucioFranco]

This is fantastic! I pulled this down locally and ran the build, everything looks good. I'm very happy this was even possible! I have a few questions but nothing blocking.

I say we

@binarylogic I think we will have to build the new image and push it to docker hub before we can merge this?

[LucioFranco]

Leaving a note here: we should open an issue for this.

cc @binarylogic

[moderation]

@LucioFranco I was just trying to compile a musl Vector on a Linux x86_64 machine with cargo build --target x86_64-unknown-linux-musl --release --verbose using Rust stable 1.36.0 and hit an openssl error. Is this problem related to this issue?

Any consideration of using Rust native rustls? See https://jbp.io/2019/07/01/rustls-vs-openssl-performance.html. This works well for rusoto https://github.com/rusoto/rusoto/blob/master/rusoto/core/README.md#usage-with-rustls

[ghost]

@moderation The OpenSSL error shouldn't be related to differences between nightly and stable Rust. Probably pkg-config is installed on your system, but returns empty CFLAGS and LIBS for openssl package. In that case setting environment variables OPENSSL_STATIC=1 and OPENSSL_INCLUDE_DIR/OPENSSL_LIB_DIR might help. See sfackler/rust-openssl#604 and https://github.com/timberio/vector/blob/master/distribution/docker/alpine/Dockerfile#L10-L13 for an example.

[moderation]

@a-rodin thanks for the tip. I'm now able to get through to the linking stage with env OPENSSL_STATIC=1 OPENSSL_INCLUDE_DIR="/usr/include" OPENSSL_LIB_DIR="/usr/lib/x86_64-linux-gnu" cargo build --target x86_64-unknown-linux-musl --release --verbose. I get error: linking with cc failed: exit code: 1 following by a large stack trace / error dump. Closer. Maybe need to wait for newer Rust releases.

[ghost]

@moderation I've seen linker crashing when building native rdkafka dependency without rdkafka/cmake_build feature, it is unrelated to Rust at all and caused by linking unused dynamic library for librdkafka. You can try cargo build --target x86_64-unknown-linux-musl --release --verbose --features="default rdkafka/cmake_build", in that case the library that makes C++ linker crash would just not be created at all.

[LucioFranco]

what are the implications of this? Is there any follow up that we might need to do here?

[ghost]

program_invocation_short_name is a global variable that contains short name of the program. It is not a part of standard C library, but a GNU extension. It is provided by musl though. The original code in LibreSSL checks on Linux whether C library is glibc and generates a compile-time error if it is not, even if program_invocation_short_name is available. With the patch the error is not thrown and program_invocation_short_name from musl is used.

Because the issue is fixed in upstream by the mentioned PR, it should be possible to remove this patching after the next release of LibreSSL.

[binarylogic]

Everything looks great here. I'm going to merge and follow up with documentation changes.