Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docker workflow updates #195

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

Docker workflow updates #195

wants to merge 17 commits into from

Conversation

mikery
Copy link
Contributor

@mikery mikery commented Mar 19, 2024

Hi @noslav and team,

This PR modifies the Docker image workflow as follows:

  • grc-image.yml and tag-release.yml are both combined into docker-image.yml
  • use docker/metadata-action to collect image metadata from GH branch/PR/tag
  • use docker/build-push-action so we can take advantage of GHA caching
  • publish images to GitHub Container Registry instead of Google Container Registry

The same workflow is used for pushes, pull requests and tags. A release is created if a v* tag is pushed. I opted to combine all of the functionality into a single workflow to keep things DRY, however you might prefer to optimise for simplicity over DRYness and have separate pipelines for branches/PRs/tags - if so please lmk and I can adjust the PR.

Image names are determined automatically based on the action that triggered the workflow:

  • push to branch main publishes image bsp-geth:main
  • pull request #n publishes image bsp-geth:pr-n
  • pushing tag v.1.2.3 publishes images bsp-geth:1.2.3, bsp-geth:1.2
  • in all cases, bsp-geth:latest is published

Using the GHA cache allows us to take full advantage of Docker layer caching which makes a significant difference in build times. This is most visible when publishing a tag - as we have already built the image for the tagged commit sha every layer of the image is a cache hit, bringing the release time down from 26 minutes to 3 minutes.

I saw some images being published to GHCR and some to GCR. Given that GCR is deprecated and will not allow creation of any new repos I opted to standardise on GHCR, but can change this if desired. "Old" Docker images for closed PRs and deleted branches will gradually accumulate in the Docker registry. I can submit a follow up PR which cleans up these images (leaving the tagged version images in place).

I have prepared similar PRs for other components with Docker images. If you are happy with this direction I can submit those PRs once any review comments on this one have been addressed.

ETA: In order for the build to succeed the GITHUB_TOKEN requires permissions to write packages.
Screenshot 2024-03-20 at 13 11 31

Copy link

codecov bot commented Mar 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 57.34%. Comparing base (90e699b) to head (4afcbb2).
Report is 17 commits behind head on main.

Current head 4afcbb2 differs from pull request most recent head 84583ed

Please upload reports for the commit 84583ed to get more accurate results.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #195      +/-   ##
==========================================
+ Coverage   57.32%   57.34%   +0.01%     
==========================================
  Files         600      600              
  Lines       80100    80100              
==========================================
+ Hits        45916    45931      +15     
+ Misses      30309    30285      -24     
- Partials     3875     3884       +9     

see 12 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90e699b...84583ed. Read the comment docs.

@noslav
Copy link
Member

noslav commented Mar 20, 2024

Hi @noslav and team,

This PR modifies the Docker image workflow as follows:

  • grc-image.yml and tag-release.yml are both combined into docker-image.yml
  • use docker/metadata-action to collect image metadata from GH branch/PR/tag
  • use docker/build-push-action so we can take advantage of GHA caching
  • publish images to GitHub Container Registry instead of Google Container Registry

The same workflow is used for pushes, pull requests and tags. A release is created if a v* tag is pushed. I opted to combine all of the functionality into a single workflow to keep things DRY, however you might prefer to optimise for simplicity over DRYness and have separate pipelines for branches/PRs/tags - if so please lmk and I can adjust the PR.

Image names are determined automatically based on the action that triggered the workflow:

  • push to branch main publishes image bsp-geth:main
  • pull request #n publishes image bsp-geth:pr-n
  • pushing tag v.1.2.3 publishes images bsp-geth:1.2.3, bsp-geth:1.2
  • in all cases, bsp-geth:latest is published

Using the GHA cache allows us to take full advantage of Docker layer caching which makes a significant difference in build times. This is most visible when publishing a tag - as we have already built the image for the tagged commit sha every layer of the image is a cache hit, bringing the release time down from 26 minutes to 3 minutes.

I saw some images being published to GHCR and some to GCR. Given that GCR is deprecated and will not allow creation of any new repos I opted to standardise on GHCR, but can change this if desired. "Old" Docker images for closed PRs and deleted branches will gradually accumulate in the Docker registry. I can submit a follow up PR which cleans up these images (leaving the tagged version images in place).

I have prepared similar PRs for other components with Docker images. If you are happy with this direction I can submit those PRs once any review comments on this one have been addressed.

ETA: In order for the build to succeed the GITHUB_TOKEN requires permissions to write packages. Screenshot 2024-03-20 at 13 11 31

@mikery This is something we can integrate and one workflow is much better than two separate ones (they were historically two separate but there's no necessity to continue in that direction esp if GCR is deprecated. Especially considering the total build time this would be great, but we cannot however cut short the build-test workflow which runs all the tests and this would still limit the total CI/CD workflow time.

One thing that isn't that clear is that even though the GITHUB_TOKEN permissions are already set to READ and WRITE for the actions settings for the repo, the built images are failing on being pushed to GHCR. Maybe just revert to older authentication type using the CR_PAT and CR_USER?

Screenshot 2024-03-20 at 11 57 28

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants