chore(operations): Add builder for armv7-unknown-linux-musleabihf

[ghost]

Closes #859

This is an early stage builder for armv7-unknown-linux-musleabihf. It is able to produce binaries, but at the moment it supports only jemallocator feature, and not leveldb and rdkafka.

The problem with leveldb is that it expects a special configure flag saying that we are in a cross-compilation environment, and leveldb-sys crate doesn't pass it to the configure script. rdkafka has some issues related to the SSL library.

Also it uses LLD as the linker, so we would need to address #1035 before merging or use ld from the GCC toolchain for ARM.

[ghost]

rdkafka and jemalloc features can be compiled now. leveldb is waiting for #1056.

[ghost]

Now we can compile Vector with all features. The resulting binary runs on Raspberry Pi 4, but cannot load config because of the same problem as in #791. So need to wait for #1035.

[LucioFranco]

A lot of this seems repeated from our other musl container? I think this would be a good point to break this out into its own script?

[ghost]

I want to combine this and the other one into a single Dockerfile when doing #946. For now the goal is to just have Vector compiled and running on the new platoform.

[LucioFranco]

Sounds good to me! I am just eager to get #946 done 😄

[ghost]

Now Vector and the tests (not integration ones) can be executed. The current status of the tests:

• Tests using grok_parser transform result in segmentation fault.
• sources::file::tests::file_max_line_bytes test fails:

  thread 'main' panicked at 'assertion failed: `(left == right)`
    left: `[Bytes(b"short"), Bytes(b"exactly 10")]`,
   right: `[Bytes(b"short"), Bytes(b"exactly 10"), Bytes(b"last short")]`', src/sources/file.rs:1117:9
  

• All other tests are passing.

[binarylogic]

Nice! Do you know how to resolve the failing test? And how can we perform this check in CI?

[ghost]

Do you know how to resolve the failing test?

The segmentation fault in grok_parser is probably caused by C bindings in onig crate using types specific to 64-bit architectures instead of generic usize, c_ptr, etc. The same might be true about the problem with file source.

And how can we perform this check in CI?

There are two ways, one is to run QEMU virtual machine emulating ARMv7 in CircleCI and another is to spawn an A1 instance from CI to run the tests. I prefer the first approach because it would allow anyone to run these tests locally and I find it more convenient to have all tests running in one place if it is possible.

[ghost]

It turns out that for statically linked binaries it is possible to just install package qemu-user on a host x86_64 system and run ARMv7 binaries in a usual way, like ./vector.

[ghost]

The file source tests are failing just because ARM devices (or QEMU emulating them) are slower than modern x86_64 machines, so sleeping 50 milliseconds is not enough. After increasing the sleep timeout up to 500 milliseconds they passed.

[binarylogic]

So close! @a-rodin what are you thinking the ETA is on this? Looking really good.

[ghost]

@binarylogic It is ready!

[binarylogic]

Nice! I know you had to branch out a lot to support this. Great work! We'll be launching marketing pages to promote this + IOT in general.

[binarylogic]

@a-rodin, @LucioFranco is out until the 25th. Would you prefer to wait, have someone else review, or go ahead and merge? I can also find someone that has more knowledge in this domain to review, but you clearly know what you're doing 😄 .

[ghost]

I think we can merge it, as it doesn’t break anything and subsequent changes in the build process would not be breaking changes.

[poupryc]

Hello. Is it planned to publish a docker image for the arm architecture?

[ghost]

@HelloEdit Yes! Issue #1196 represents it.