Use packer, or something else, to efficiently build Vector's CI Docker images

[binarylogic]

Overview

Currently, we have a hacked together script for building all of the Docker images necessary to execute Vector's CI process via make build-ci-docker-images:

https://github.com/timberio/vector/blob/master/scripts/build-ci-docker-images.sh

And you can see all of the CI Dockerfiles here:

https://github.com/timberio/vector/tree/master/scripts/ci-docker-images

This is a pretty heavy process to run locally. In fact, I can't get the x86_64-unknown-linux-musl image to build locally. I know @LucioFranco had the same problem and resolved this by building the images on a 96 core EC2 instance.

Solution Shaping

I think it might be wise to build these images with something like packer, where we can build them in parallel on more powerful remote machines. This also takes care of publishing and so on.

Vector Infrastructure

Broadly, there have been talks of being able to rely on remote services for the Vector test suite as well, so I'm wondering if there is a more generic way of managing infrastructure for Vector that would cover all cases. Specifically, @LucioFranco has complained that the number of Docker images necessary to run the Vector tests "slows his computer to a halt". Perhaps we could provide a way for a developer to provision personal resources that the tests could interact with? For example, an Ansible script could be used to provision all of the necessary resources and the developer could configure Ansible's inventory to use the localhost or remote resources. (thinking out loud).

[binarylogic]

Noting, all images build fine locally except for the *-musl image, which should be resolved by #946.

[binarylogic]

Closing this in favor of #946