Upgrade dependencies 2024-03-18 (#6064, #6072)

[achave11-ucsc]

Connected issue: #6064, #6072

Checklist

Author

• Target branch is develop
• Name of PR branch matches upgrades/yyyy-mm-dd
• On ZenHub, PR is connected to the upgrade issue it resolves
• PR title matches Upgrade dependencies yyyy-mm-dd
• PR title references the connected issue

Author (upgrading deployments)

• Documented upgrading of deployments in UPGRADING.rst _{or this PR does not require upgrading deployments}
• Added u tag to commit title _{or this PR does not require upgrading deployments}
• Ran make image_manifests.json and committed the resulting changes _{or this PR does not modify azul_docker_images, or any other variables referenced in the definition of that variable}
• This PR is labeled upgrade _{or does not require upgrading deployments}
• This PR is labeled deploy:shared _{or does not modify image_manifests.json, and does not require deploying the shared component for any other reason}
• This PR is labeled deploy:gitlab _{or does not require deploying the gitlab component}
• This PR is labeled deploy:runner _{or does not require deploying the runner image}

Author (before every review)

• Rebased PR branch on develop, squashed old fixups
• Ran make requirements_update _{or this PR does not modify requirements*.txt, common.mk, Makefile and Dockerfile}
• Added R tag to commit title _{or this PR does not modify requirements*.txt}
• This PR is labeled reqs _{or does not modify requirements*.txt}
• make integration_test passes in personal deployment _{or this PR does not modify functionality that could affect the IT outcome}

System administrator (after approval)

• Actually approved the PR
• Labeled connected issue as no demo
• A comment to this PR details the completed security design review _{or this PR is a promotion or a backport}
• PR title is appropriate as title of merge commit
• Moved ticket to Approved column
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Squashed PR branch and rebased onto develop
• Sanity-checked history
• Pushed PR branch to GitHub
• Ran _select dev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select dev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvildev.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Ran _select anvilprod.shared && CI_COMMIT_REF_NAME=develop make -C terraform/shared apply_keep_unused _{or this PR is not labeled deploy:shared}
• Ran _select anvilprod.gitlab && CI_COMMIT_REF_NAME=develop make -C terraform/gitlab apply _{or this PR is not labeled deploy:gitlab}
• Checked the items in the next section _{or this PR is labeled deploy:gitlab}
• Assigned system administrator _{or this PR is not labeled deploy:gitlab}

System administrator

• Background migrations for dev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvildev.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• Background migrations for anvilprod.gitlab are complete _{or this PR is not labeled deploy:gitlab}
• PR is assigned to only the operator

Operator (before pushing merge the commit)

• Ran _select dev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvildev.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Ran _select anvilprod.gitlab && make -C terraform/gitlab/runner _{or this PR is not labeled deploy:runner}
• Added sandbox label
• Pushed PR branch to GitLab dev
• Pushed PR branch to GitLab anvildev
• Pushed PR branch to GitLab anvilprod
• Build passes in sandbox deployment
• Build passes in anvilbox deployment
• Build passes in hammerbox deployment
• Reviewed build logs for anomalies in sandbox deployment
• Reviewed build logs for anomalies in anvilbox deployment
• Reviewed build logs for anomalies in hammerbox deployment
• The title of the merge commit starts with the title of this PR
• Added PR # reference to merge commit title
• Collected commit title tags in merge commit title _{but excluded any p tags}
• Moved connected issue to Merged column in ZenHub
• Pushed merge commit to GitHub

Operator (after pushing the merge commit)

• Pushed merge commit to GitLab dev
• Pushed merge commit to GitLab anvildev
• Pushed merge commit to GitLab anvilprod
• Build passes on GitLab dev
• Reviewed build logs for anomalies on GitLab dev
• Build passes on GitLab anvildev
• Reviewed build logs for anomalies on GitLab anvildev
• Build passes on GitLab anvilprod
• Reviewed build logs for anomalies on GitLab anvilprod
• Ran _select dev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvildev.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Ran _select anvilprod.shared && make -C terraform/shared apply _{or this PR is not labeled deploy:shared}
• Deleted PR branch from GitHub
• Deleted PR branch from GitLab dev
• Deleted PR branch from GitLab anvildev
• Deleted PR branch from GitLab anvilprod

Operator

• Ran script/export_inspector_findings.py against anvilprod, imported results to Google Sheet and posted screenshot of relevant¹ findings as a comment on the connected issue.
• Propagated the deploy:shared, deploy:gitlab, deploy:runner, reindex:partial and reindex:prod labels to the next promotion PR _{or this PR carries none of these labels}
• Propagated any specific instructions related to the deploy:shared, deploy:gitlab, deploy:runner, reindex:partial and reindex:prod labels from the description of this PR to that of the next promotion PR _{or this PR carries none of these labels}
• PR is assigned to only the system administrator

¹A relevant finding is a high or critical vulnerability in an image
that is used within the security boundary. Images not used within the boundary
are tracked in azul.docker_images under a key starting with _.

System administrator

• No currently reported vulnerability requires immediate attention
• PR is assigned to no one

Shorthand for review comments

• L line is too long
• W line wrapping is wrong
• Q bad quotes
• F other formatting problem

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 85.33%. Comparing base (be78a54) to head (696f620).

Additional details and impacted files

@@           Coverage Diff            @@
##           develop    #6078   +/-   ##
========================================
  Coverage    85.33%   85.33%           
========================================
  Files          154      154           
  Lines        19969    19969           
========================================
+ Hits         17040    17041    +1     
+ Misses        2929     2928    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[coveralls]

coverage: 85.355% (+0.005%) from 85.35%
when pulling 696f620 on upgrades/2024-03-18
into be78a54 on develop.

[achave11-ucsc]

Upgrading the instance AMI to the latest version (v3.0.0.1) prevents the OpenSSH server demon from starting successfully in addition to the GitLab instance coming back online appropriately.

[achave11-ucsc]

Latest version of ClamAV (1.3.0-43) continues to fail the scan with a segfault error:

$ sudo /usr/bin/docker run --name clamscan --rm --volume /var/run/docker.sock:/var/run/docker.sock --volume /:/scan:ro --volume /mnt/gitlab/clamav:/var/lib/clamav:rw 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/clamav/clamav:1.3.0-43 /bin/sh -c "freshclam && echo freshclam succeeded || (echo freshclam failed; false) && clamscan --verbose --recursive --infected --allmatch=yes /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial && echo clamscan succeeded || (echo clamscan failed; false)"
Unable to find image '122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/clamav/clamav:1.3.0-43' locally
1.3.0-43: Pulling from clamav/clamav
4abcf2066143: Already exists 
ca21dcf46d09: Pull complete 
dc5692c0f3c1: Pull complete 
dd05b9175ba5: Pull complete 
e0be838d51cc: Pull complete 
dbff6d342e2f: Pull complete 
a13c85e506db: Pull complete 
Digest: sha256:866bbd5f1f4871773c67f533fb0114e2657143f1a90d8f860bc72dcd9f49dc96
Status: Downloaded newer image for 122796619775.dkr.ecr.us-east-1.amazonaws.com/docker.io/clamav/clamav:1.3.0-43
ClamAV update process started at Thu Mar 21 01:49:59 2024
daily.cld database is up-to-date (version: 27220, sigs: 2055792, f-level: 90, builder: raynman)
main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
bytecode.cld database is up-to-date (version: 335, sigs: 86, f-level: 90, builder: raynman)
freshclam succeeded
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/grep.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/urllibcompat.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/formatter.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/scmposix.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/mergestate.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/metadata.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/exthelper.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/templatefilters.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/vfs.py
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/el/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/da/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/sv/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ru/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ro/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/de/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_CN/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/zh_TW/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/it/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/fr/LC_MESSAGES/hg.mo
Scanning /scan/mnt/gitlab/docker/overlay2/cb91ee0a392a8b157d52474fa67092b0f65041470a53a5db9824b6d47461187a/diff/usr/lib/python3/dist-packages/mercurial/locale/ja/LC_MESSAGES/hg.mo
Segmentation fault (core dumped)
clamscan failed

[achave11-ucsc]

@hannes-ucsc, also note I've assigned you the issue to complete sys admin related CL items: #6064 (comment)

[hannes-ucsc]

Security design review

• Security design review completed; this PR does not …
  • … affect authentication; for example:
    • OAuth 2.0 with the application (API or Swagger UI)
    • Authentication of developers with Google Cloud APIs
    • Authentication of developers with AWS APIs
    • Authentication with a GitLab instance in the system
    • Password and 2FA authentication with GitHub
    • API access token authentication with GitHub
    • Authentication with Terra
  • … affect the permissions of internal users like access to
    • Cloud resources on AWS and GCP
    • GitLab repositories, projects and groups, administration
    • an EC2 instance via SSH
    • GitHub issues, pull requests, commits, commit statuses, wikis, repositories, organizations
  • … affect the permissions of external users like access to
    • TDR snapshots
  • … affect permissions of service or bot accounts
    • Cloud resources on AWS and GCP
  • … affect audit logging in the system, like
    • adding, removing or changing a log message that represents an auditable event
    • changing the routing of log messages through the system
  • … affect monitoring of the system
  • … introduce a new software dependency like
    • Python packages on PYPI
    • Command-line utilities
    • Docker images
    • Terraform providers
  • … add an interface that exposes sensitive or confidential data at the security boundary
  • … affect the encryption of data at rest
  • … require persistence of sensitive or confidential data that might require encryption at rest
  • … require unencrypted transmission of data within the security boundary
  • … affect the network security layer; for example by
    • modifying, adding or removing firewall rules
    • modifying, adding or removing security groups
    • changing or adding a port a service, proxy or load balancer listens on
• Documentation on any unchecked boxes is provided in comments below

[hannes-ucsc]

@hannes-ucsc, also note I've assigned you the issue to complete sys admin related CL items: #6064 (comment)

#6064 (comment)