chore(operations)!: Teach Environment to build for other architectures.

[Hoverbear]

Closes #3311

This introduces a simplified cross-compiler toolchain for us.

After removing our Armv7 builds in #3352 we could opt for a much, much simpler option with https://github.com/richfelker/musl-cross-make. This introduces the components to our environment, as well as adds tooling sufficient for our contributors to use it locally on most Linux machines.

Please note this process is a tad rough around the edges still, I plan to touch it up over the coming months. :)

As a consequence of #2824 we needed to teach our Environment to be multi-distro. I added this as well, it currently acts as CentOS7 during gnu builds, and Ubuntu 20.04 for MUSL builds.

Notable Changes

• make build-{x86_64, aarch64}-* are altered and now build outside of docker containers by default. They use musl-cross-make and you can use ENVIRONMENT=true to run them in our friendly environment.
• make build-armv7-* is gone as per chore(operations)!: Removal of Armv7 support #3352
• I cleaned up make check-examples since it was failing.
• You can now specify ENVIRONMENT_BASEIMAGE with a docker container (default ubuntu:20.04). The environment will try to build from that baseimage by searching the scripts/environments/bootstrap/ folder. You can also override with ENVIRONMENT_BOOTSTRAP set to a shell script.
• This increases our podman support. I think these should work!
• The new cross toolchains default to scripts/environment/cross which is part of our .gitignore. You can set a different place with MUSL_CROSS_MAKE environment variable. I personally cloned musl-cross-make into my normal Git trees and set this environment variable.
• CI jobs are added for build passes of this new process. This will be expanded later to include things like testing. I expect all our test and check jobs will collapse down into these new jobs in the future.
• A RELEASE flag defaulting to true is available. Use RELEASE=false to dramatically speed up build-* jobs.

Important notes

• We originally talked about storing the toolchains in target but this lead to a poor user experience. Crosschains take up to 15 minutes to build so it's better to keep them outside of this folder we frequently wipe.
• We have long term goals to replace our nightly and release workflows with per-commit/PR release flows that are published when we tag. This is pulling us in that direction!

Remaining burndown

• Docs
• Ensure CI is green
• Explore testing builds.
• Clean up UX. (Warn if doing a cross build without the toolchain, warn if doing builds not on linux)

[Hoverbear]

Giving this a test in our docker environment

[Hoverbear]

I'm pulling in the ideas of #3257 here

[Hoverbear]

Also pulled in #3352

[Hoverbear]

I did some feature flag tweaking so I'm running a full build suite locally as well as up here on CI. :)

[Hoverbear]

rbenv is struggling because the ubuntu distro has homebrew installed...

[Hoverbear]

This is now working for me locally!

[binarylogic]

@Hoverbear can you link this to the issues it closes?

[Hoverbear]

@binarylogic This would close #2971 and part of #3261

It'd also accomplish #2899's original goal. It's also needed before we do #3125 #3145 #1301 #3124 #3123 #3122 as well as would allow us to restore #3142

[Hoverbear]

So this mostly works.

All the makefile integration and the vast majority of the building works.

As far as I can tell, the problem lies in the Glibc to Musl linking.

So we need to support the following goals:

• Centos:7 base to glibc x86 build.
• Anything to musl x86 build. (We use Ubuntu and musl-cross-make here)
• Anything to musl aarch64 build. (We use Ubuntu and musl-cross-make here)

Further goals are out of scope at this time, but this style of approach allows for our environment to learn to cross compile to other architectures in the future.

We also looked at using docker buildx in #3257 but it conflicts with our goal in #781 .

So when you build the musl builds, there's a problem that (I think) sasl2 or rdkafka's linkage where it's getting built with the wrong libc.

I am fairly certain that this is a result of us doing something wrong with the C++ related cross compiling variables, as I am not very experienced here. As far as I know, all the Rust and C, and the majority of the C++, is working right, but it seems one thing is not.

[Hoverbear]

We're opting for #3657