Discussion around which OSs to use in Static Docker containers

[Haroon-Khel]

It would be helpful to discuss and create a policy in regards to which operating systems we use for our static docker containers. Such policy would specify things such as

• When to add new OS versions to the inventory
• When to replace 'outdated' OS versions
• How many containers to host on each dockerhost machine
  etc

Such a policy would help us to have the latest OSs as part of our static docker inventory.

I also like the idea of an automated way of setting up new static docker containers, perhaps via awx, and an automated way of removing ones we no longer need

[sxa]

Noting that https://github.com/adoptium/infrastructure/wiki/End-of-support-date-for-OS-distributions is intended to be used as input to these sorts of decisions for docker and non-docker systems.

Semi-related: #3237

[Haroon-Khel]

We should also include in this policy a cut off point after which we replace containers running OSs that are not LTS. test-docker-ubuntu2010-armv8l-2 is an example. 20.10 is not a Ubuntu LTS and therefore its repositories are not maintained after EOL.

ref #2662 (comment)

My attempts to install fontconfig to solve a test error

Err:1 http://ports.ubuntu.com/ubuntu-ports groovy/main arm64 fonts-dejavu-core all 2.37-2
  404  Not Found [IP: 185.125.190.39 80]
E: Failed to fetch http://ports.ubuntu.com/ubuntu-ports/pool/main/f/fonts-dejavu/fonts-dejavu-core_2.37-2_all.deb  404  Not Found [IP: 185.125.190.39 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
...
root@93d2b4e13a22:~# apt-get update
Ign:1 http://ports.ubuntu.com/ubuntu-ports groovy InRelease
Ign:2 http://ports.ubuntu.com/ubuntu-ports groovy-updates InRelease
Ign:3 http://ports.ubuntu.com/ubuntu-ports groovy-backports InRelease
Ign:4 http://ports.ubuntu.com/ubuntu-ports groovy-security InRelease
Err:5 http://ports.ubuntu.com/ubuntu-ports groovy Release
  404  Not Found [IP: 185.125.190.39 80]
Err:6 http://ports.ubuntu.com/ubuntu-ports groovy-updates Release
  404  Not Found [IP: 185.125.190.39 80]
Err:7 http://ports.ubuntu.com/ubuntu-ports groovy-backports Release
  404  Not Found [IP: 185.125.190.39 80]
Err:8 http://ports.ubuntu.com/ubuntu-ports groovy-security Release
  404  Not Found [IP: 185.125.190.39 80]
Reading package lists... Done
E: The repository 'http://ports.ubuntu.com/ubuntu-ports groovy Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports groovy-updates Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports groovy-backports Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: The repository 'http://ports.ubuntu.com/ubuntu-ports groovy-security Release' no longer has a Release file.
N: Updating from such a repository can't be done securely, and is therefore disabled by default.
N: See apt-secure(8) manpage for repository creation and user configuration details. 

[sxa]

We should also include in this policy a cut off point after which we replace containers running OSs that are not LTS. test-docker-ubuntu2010-armv8l-2 is an example. 20.10 is not a Ubuntu LTS and therefore its repositories are not maintained after EOL.

Yeah that's fine with me - if you want to shut those down instead of fixing them it's very much a 👍🏻 from me. IMHO the ideal thing to do would be nice to be able to keep updating it to the "current" STS release if available, so we're always testing on the latest supported.

[sxa]

As mentioned earlier this week, the Node.js project uses a syntax in the inventory file which has a containers stanza against each machine, like this (The file isn't publicly visible because of the secret entries in it, so here's a snipped with those bits removed:

        ubuntu1804_docker-x64-1:
          server_jobs: 4
          containers:
            - { name: 'test-digitalocean-alpine311_container-x64-1', os: 'alpine311', secret: 'xxxx' }
            - { name: 'test-digitalocean-alpine312_container-x64-1', os: 'alpine312', secret: 'xxxx' }
            - { name: 'test-digitalocean-rhel8_arm_cross_container-x64-1', os: 'rhel8_arm_cross', secret: 'xxxx' }

Those secret entries are used to set up the containers to call back into the jenkins server instead of exposing the containers over ssh. The containers are built on-the-fly during the playbook run if there is a containers section - you can see the code for that in https://github.com/nodejs/build/blob/94ff7b6ae093ba794d6d04ad01d0e4ddf89b3c45/ansible/roles/docker/tasks/main.yml#L178

[Haroon-Khel]

Continued in #3368