Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build Docker image and push to GHCR #230

Open
wants to merge 19 commits into
base: unstable/v1
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
54 changes: 54 additions & 0 deletions .github/workflows/build-and-push-docker-image.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
---

name: 🏗️

on: # yamllint disable-line rule:truthy
pull_request:
push:
branches: ["release/*", "unstable/*"]
workflow_dispatch:
inputs:
tag:
description: Docker image tag
Comment on lines +11 to +12
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we incorporate the prior art from the discussions in #45 and accept the 3-segment version as an input, then deduce everything else from that + push the Git tag and advance the proper branches? Perhaps, this could also happen before this PR so it exists as a separate base.

I don't understand what you're asking here. You want separate inputs for major, minor, and patch versions? What if you're building the Docker image from the release/v1 branch? What is the "3-segment version" in that case?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland I was talking about the workflow inputs. Everything else should be computed from them.

The current workflow for me is as follows:

  1. Tag v1.10.1 on top of unstable/v1
  2. Fast-forward release/v1 to v1.10.1
  3. Fast-forward release/v1.10 to v1.10.1 (or create it)
  4. git push --atomic everything in one go
  5. Create a GitHub Release from the UI

With this, my 3-segment “input” is 1.10.1 and v1+v1.10 are being extracted from that. Do you think we could bake containers into this flow with not many changes? This will involve tagging the same image with multiple tags, as I understand. And the end-users referencing versions by both tags and branches should be able to retrieve whatever container there is, right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@webknjaz thanks for explaining that. I think we're actually pretty close to the desired workflow. There's just a small update needed to the Docker build workflow so it pushes tags with the major and minor version numbers. I've pushed that change.

Your updated workflow would look like this:

  1. Tag v1.10.1 on top of unstable/v1
  2. Fast-forward release/v1 to v1.10.1
  3. Fast-forward release/v1.10 to v1.10.1 (or create it)
  4. git push --atomic everything in one go
  5. Trigger the 🏗️ workflow from the Git tag. The workflow will push the Docker image with three Docker tags:
    1. ghcr.io/pypa/gh-action-pypi-publish:v1.10.1
    2. ghcr.io/pypa/gh-action-pypi-publish:v1.10
    3. ghcr.io/pypa/gh-action-pypi-publish:v1
  6. Create a GitHub Release from the UI

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland so with #45, I wanted to move all these steps inside the automation. For publishing Python projects, I usually use a workflow with workflow_dispatch that allows me to type in the desired version to release and that workflow creates the git tag, the gh release, signs stuff and so on.

The idea is that tag+branches+gh-release represent the result of release automation being successful and aren't triggers or manual actions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland so with #45, I wanted to move all these steps inside the automation. For publishing Python projects, I usually use a workflow with workflow_dispatch that allows me to type in the desired version to release and that workflow creates the git tag, the gh release, signs stuff and so on.

The idea is that tag+branches+gh-release represent the result of release automation being successful and aren't triggers or manual actions.

So... you want the Docker build workflow to run automatically when a tag is created? Or when a GitHub Release is created?

Please be more specific here, and please limit the scope of your requests. The goal of this PR is "Build Docker image and push to GHCR." The goal is not to develop your entire release workflow.

PR #45 is currently a draft and I don't think I should have to make this PR dependent on some other draft PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So... you want the Docker build workflow to run automatically when a tag is created? Or when a GitHub Release is created?

No to both. In my release processes, tags + GH Releases are the results of publishing, not the triggers. So when I trigger workflow_dispatch (which is a button click on GH UI + the desired version entered into the input field), the workflows create what I want to publish, send that to the primary platform (PyPI in case of Python projects or a container registry in the context of this specific PR), and only after that the same workflow creates the Git tag and the GH Release.

Please be more specific here, and please limit the scope of your requests. The goal of this PR is "Build Docker image and push to GHCR." The goal is not to develop your entire release workflow.

PR #45 is currently a draft and I don't think I should have to make this PR dependent on some other draft PR.

I only pointed to that PR as this one seems to be touching the same topic partially. I was thinking that since you're already making a workflow that publishes the container, that could piggyback on top of this effort. I wasn't thinking of it as a dependency but rather as a replacement, but wanted to point to it for the pre-existing context around what I wanted to implement for quite a while.

This PR is attempting to change my release workflow already, after all. So I wanted to make sure it doesn't go in the direction that would complicate things, given that I already envision the release process to develop in a certain way.

If it were possible to merge the PR and keep the workflow where the Git tag is pushed, and it works right away without a container published for a while, it'd be where we could scope it. However, AFAIU, this will no longer work after merging since when there's a tag and the end-users start using it, the action will attempt pulling the image that is not yet there. There's a race condition in this process, meaning that it is prone to human error and needs to be handled carefully. And that's why I was thinking that it'd be good to integrate all the other things in a way that would keep the process robust.

P.S. Would it be useful to still give the end-users some way to force building the container instead of pulling the cached one?

required: true
type: string

jobs:
smoke-test:
uses: ./.github/workflows/reusable-smoke-test.yml
build-and-push:
if: github.event_name != 'pull_request'
runs-on: ubuntu-latest
needs:
- smoke-test
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- name: Build Docker image
run: |
DOCKER_TAG="${DOCKER_TAG/'/'/'-'}"
DOCKER_TAG_MAJOR=$(echo "$DOCKER_TAG" | cut -d '.' -f 1)
DOCKER_TAG_MAJOR_MINOR=$(echo "$DOCKER_TAG" | cut -d '.' -f 1-2)
IMAGE="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG}"
IMAGE_MAJOR="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG_MAJOR}"
IMAGE_MAJOR_MINOR="ghcr.io/$GITHUB_REPOSITORY:${DOCKER_TAG_MAJOR_MINOR}"
echo "IMAGE=$IMAGE" >>"$GITHUB_ENV"
echo "IMAGE_MAJOR=$IMAGE_MAJOR" >>"$GITHUB_ENV"
echo "IMAGE_MAJOR_MINOR=$IMAGE_MAJOR_MINOR" >>"$GITHUB_ENV"
docker build . \
--build-arg BUILDKIT_INLINE_CACHE=1 \
--cache-from $IMAGE \
--tag $IMAGE
docker tag $IMAGE $IMAGE_MAJOR
docker tag $IMAGE $IMAGE_MAJOR_MINOR
env:
DOCKER_TAG: ${{ inputs.tag || github.ref_name }}
- name: Log in to GHCR
run: >-
echo ${{ secrets.GITHUB_TOKEN }} |
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
docker login ghcr.io -u $GITHUB_ACTOR --password-stdin
- name: Push Docker image to GHCR
run: |
docker push $IMAGE
docker push $IMAGE_MAJOR
docker push $IMAGE_MAJOR_MINOR
Original file line number Diff line number Diff line change
@@ -1,10 +1,9 @@
---

name: 🧪
name: ♻️ 🧪

on: # yamllint disable-line rule:truthy
push:
pull_request:
workflow_call:

env:
devpi-password: abcd1234
Expand All @@ -27,7 +26,33 @@ env:
PYTEST_THEME_MODE

jobs:
fail-fast:

strategy:
matrix:
os: [macos-latest, windows-latest]

runs-on: ${{ matrix.os }}

timeout-minutes: 2

steps:
- name: Check out the action locally
uses: actions/checkout@v4
with:
path: test
- name: Fail-fast in unsupported environments
continue-on-error: true
id: fail-fast
uses: ./test
- name: Error if action did not fail-fast in unsupported environments
if: steps.fail-fast.outcome == 'success'
run: |
>&2 echo This action should fail-fast in unsupported environments.
exit 1

smoke-test:

runs-on: ubuntu-latest

services:
Expand Down
79 changes: 67 additions & 12 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -91,15 +91,70 @@ branding:
color: yellow
icon: upload-cloud
runs:
using: docker
image: Dockerfile
args:
- ${{ inputs.user }}
- ${{ inputs.password }}
- ${{ inputs.repository-url }}
- ${{ inputs.packages-dir }}
- ${{ inputs.verify-metadata }}
- ${{ inputs.skip-existing }}
- ${{ inputs.verbose }}
- ${{ inputs.print-hash }}
- ${{ inputs.attestations }}
using: composite
steps:
br3ndonland marked this conversation as resolved.
Show resolved Hide resolved
- name: Fail-fast in unsupported environments
if: runner.os != 'Linux'
run: |
>&2 echo This action is only able to run under GNU/Linux environments
exit 1
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
shell: bash -eEuo pipefail {0}
- name: Reset path if needed
run: |
# Reset path if needed
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
# https://github.com/pypa/gh-action-pypi-publish/issues/112
if [[ $PATH != *"/usr/bin"* ]]; then
echo "\$PATH=$PATH. Resetting \$PATH for GitHub Actions."
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
echo "PATH=$PATH" >>"$GITHUB_ENV"
echo "$PATH" >>"$GITHUB_PATH"
echo "\$PATH reset. \$PATH=$PATH"
fi
shell: bash
- name: Set repo and ref from which to run Docker container action
id: set-repo-and-ref
run: |
# Set repo and ref from which to run Docker container action
# to handle cases in which `github.action_` context is not set
# https://github.com/actions/runner/issues/2473
REF=${{ env.ACTION_REF || env.PR_REF || github.ref_name }}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@br3ndonland when running in another repo, wouldn't github.ref_name point to that repo instead of ours? Is it safe to use this var as a fallback? Same question for github.event.pull_request. Is this to cover contributions to the action itself?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If another repo is using the marketplace action (uses: pypa/gh-action-pypi-publish), env.ACTION_REF will be set from github.action_ref.

If the action is being run on a PR to this repo, env.PR_REF will be set from github.event.pull_request.head.ref.

If the action is being run from a push, release, or other event in this repo, github.ref_name will be used.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright. Thanks for clarifying. So I'll need to be careful in the release process to make an image before tagging…

REPO=${{ env.ACTION_REPO || env.PR_REPO || github.repository }}
REPO_ID=${{ env.PR_REPO_ID || github.repository_id }}
echo "ref=$REF" >>"$GITHUB_OUTPUT"
echo "repo=$REPO" >>"$GITHUB_OUTPUT"
echo "repo-id=$REPO_ID" >>"$GITHUB_OUTPUT"
shell: bash
env:
ACTION_REF: ${{ github.action_ref }}
ACTION_REPO: ${{ github.action_repository }}
PR_REF: ${{ github.event.pull_request.head.ref }}
PR_REPO: ${{ github.event.pull_request.head.repo.full_name }}
PR_REPO_ID: ${{ github.event.pull_request.base.repo.id }}
- name: Check out action repo
uses: actions/checkout@v4
with:
path: action-repo
ref: ${{ steps.set-repo-and-ref.outputs.ref }}
repository: ${{ steps.set-repo-and-ref.outputs.repo }}
- name: Create Docker container action
run: |
# Create Docker container action
python create-docker-action.py
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry if I'm missing something here: why is creating the container action done dynamically like this? Is there a reason it can't be a static file?

Copy link
Contributor Author

@br3ndonland br3ndonland Jun 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the reason is that otherwise there's no way to specify the correct Docker tag.

Docker actions support pulling in pre-built Docker images by supplying a registry address to the image: key. The downside to this syntax is that there's no way to specify the correct Docker tag because the GitHub Actions image: and uses: keys don't accept any context. For example, if a user's workflow has uses: pypa/gh-action-pypi-publish@release/v1.8, then the action should pull in a Docker image built from the release/v1.8 ref, something like ghcr.io/pypa/gh-action-pypi-publish:release-v1.8 (Docker tags can't have /).

# this works but the image tag can't be customized
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:release-v1.8
# this doesn't work because `image:` doesn't support context
runs:
  using: docker
  image: docker://ghcr.io/pypa/gh-action-pypi-publish:${{ github.action_ref }}

The workaround is to switch the top-level action.yml to a composite action that then creates the Docker container action, substituting the correct image name and tag.

Originally this PR proposed to create the Docker container action as a YAML file with a single placeholder field, like image: {{image}}, then replace the {{image}} placeholder with the image tag using sed. The maintainer preferred to have the entire Docker container action generated with a Python script instead.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@woodruffw I requested that. I don't want to have another file with hard-coded contents in the repo. There was a template-like thing before. The published container version is being generated dynamically.

env:
REF: ${{ steps.set-repo-and-ref.outputs.ref }}
REPO: ${{ steps.set-repo-and-ref.outputs.repo }}
REPO_ID: ${{ steps.set-repo-and-ref.outputs.repo-id }}
shell: bash
working-directory: action-repo
- name: Run Docker container
uses: ./action-repo/.github/actions/run-docker-container
with:
user: ${{ inputs.user }}
password: ${{ inputs.password }}
repository-url: ${{ inputs.repository-url || inputs.repository_url }}
packages-dir: ${{ inputs.packages-dir || inputs.packages_dir }}
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
verify-metadata: ${{ inputs.verify-metadata || inputs.verify_metadata }}
skip-existing: ${{ inputs.skip-existing || inputs.skip_existing }}
verbose: ${{ inputs.verbose }}
print-hash: ${{ inputs.print-hash || inputs.print_hash }}
attestations: ${{ inputs.attestations }}
75 changes: 75 additions & 0 deletions create-docker-action.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
import json
import os
import pathlib

DESCRIPTION = 'description'
REQUIRED = 'required'

REF = os.environ['REF']
REPO = os.environ['REPO']
REPO_ID = os.environ['REPO_ID']
REPO_ID_GH_ACTION = '178055147'


def set_image(ref: str, repo: str, repo_id: str) -> str:
if repo_id == REPO_ID_GH_ACTION:
return '../../../Dockerfile'
docker_ref = ref.replace('/', '-')
return f'docker://ghcr.io/{repo}:{docker_ref}'


image = set_image(REF, REPO, REPO_ID)

action = {
'name': '🏃',
DESCRIPTION: (
'Run Docker container to upload Python distribution packages to PyPI'
),
'inputs': {
webknjaz marked this conversation as resolved.
Show resolved Hide resolved
'user': {DESCRIPTION: 'PyPI user', REQUIRED: False},
'password': {
DESCRIPTION: 'Password for your PyPI user or an access token',
REQUIRED: False,
},
'repository-url': {
DESCRIPTION: 'The repository URL to use',
REQUIRED: False,
},
'packages-dir': {
DESCRIPTION: 'The target directory for distribution',
REQUIRED: False,
},
'verify-metadata': {
DESCRIPTION: 'Check metadata before uploading',
REQUIRED: False,
},
'skip-existing': {
DESCRIPTION: (
'Do not fail if a Python package distribution'
' exists in the target package index'
),
REQUIRED: False,
},
'verbose': {DESCRIPTION: 'Show verbose output.', REQUIRED: False},
'print-hash': {
DESCRIPTION: 'Show hash values of files to be uploaded',
REQUIRED: False,
},
'attestations': {
DESCRIPTION: (
'[EXPERIMENTAL]'
' Enable experimental support for PEP 740 attestations.'
' Only works with PyPI and TestPyPI via Trusted Publishing.'
),
REQUIRED: False,
},
},
'runs': {
'using': 'docker',
'image': image,
},
}

action_path = pathlib.Path('.github/actions/run-docker-container/action.yml')
action_path.parent.mkdir(parents=True, exist_ok=True)
action_path.write_text(json.dumps(action, ensure_ascii=False), encoding='utf-8')
Loading