Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

use reusable workflows in create-release workflow. #454

Closed
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
a34787a
add a reusable release workflow
friedrichwilken Jan 31, 2024
99654f5
add scripts
friedrichwilken Feb 1, 2024
d0f972c
remove old script
friedrichwilken Feb 1, 2024
deb19cb
add new needs
friedrichwilken Feb 1, 2024
655263b
remove obsolete scripts
friedrichwilken Feb 1, 2024
7dd2fac
fix spelling
friedrichwilken Feb 1, 2024
cd49f77
pin workflows to commit sha
friedrichwilken Feb 1, 2024
a78a69b
fix typo
friedrichwilken Feb 1, 2024
7c15fad
rename manifest files
friedrichwilken Feb 1, 2024
29df00a
Update hack/scripts/create_changelog.sh
friedrichwilken Feb 1, 2024
2698b0d
fix typos in create-changelog
friedrichwilken Feb 2, 2024
01c0742
simplify variables
friedrichwilken Feb 2, 2024
336e3c7
Update hack/scripts/render-sec-scanners-config.sh
friedrichwilken Feb 2, 2024
4491d35
new to authors
friedrichwilken Feb 2, 2024
5eb07cb
remove removal of temp files
friedrichwilken Feb 2, 2024
f5c245a
update bump-sec-scanners-config sha
friedrichwilken Feb 7, 2024
78e1ae7
add persmissions to scripts
friedrichwilken Feb 7, 2024
98b0ee2
update sha for trigger-prow-build-job
friedrichwilken Feb 7, 2024
5738fa5
bump trigger-prow-build-job
friedrichwilken Feb 7, 2024
3f88f1e
change gh token
friedrichwilken Feb 7, 2024
440aafa
auto-bump sec-scanners-config: 2024-02-07-10-07-42 (#464)
kyma-eventing-bot Feb 7, 2024
51d7832
change token
friedrichwilken Feb 7, 2024
9235769
inherit secrets to trigger-prow=build-job-reusable
friedrichwilken Feb 7, 2024
6c27b43
bump create-draft-release
friedrichwilken Feb 7, 2024
fe5e1bc
bump sha for create draft release
friedrichwilken Feb 7, 2024
302370e
bump redener manifests
friedrichwilken Feb 7, 2024
3720d1f
use old create_changelog.sh
friedrichwilken Feb 8, 2024
0b0d1e1
add step to update kusomization
friedrichwilken Feb 8, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 65 additions & 83 deletions .github/workflows/create-release.yml
Original file line number Diff line number Diff line change
@@ -1,96 +1,78 @@
name: "Create release"
name: Create release

on:
workflow_dispatch:
inputs:
DRY_RUN:
type: boolean
default: false
description: "Dry run: setting this to `true` will skip the final step of publishing the release, leaving it as a draft-release. This can be helpful for testing and debugging. Note that the tag and release are still getting generated and the next run of this result in incremented release version. To avoid this, do not forget to delete the resulting release and tag."

jobs:
verify-release:
name: Verify release
runs-on: ubuntu-latest
outputs:
version: ${{ steps.gen-version.outputs.VERSION }}
gen-version:
name: Generate semantic version from branch and tags
uses: kyma-project/eventing-tools/.github/workflows/get-version-from-release-branch-reusable.yml@main
# outputs: VERSION
Copy link
Contributor

@marcobebway marcobebway Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this always return the workflow yaml from main~head? If yes, can we use a commit SHA instead, just in case main~head gets broken for some reason.

Note: This also applies to other places in this PR using @main.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, I am happy that you ask about this because I was also between these two options.
Sticking this to a commit sha makes this less susceptible to bug, so yes, let's do that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed it to a commit sha everywhere.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to update the create-draft-release workflow sha, after this PR got merged.


steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
bump-sec-scanners-config:
name: Bump the sec-scandners-config.yaml
needs: gen-version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
name: Bump the sec-scandners-config.yaml
name: Bump the sec-scanners-config.yaml

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed!

uses: kyma-project/eventing-tools/.github/workflows/bump-sec-scanners-config-reusable.yml@main
with:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) ignore for now, after noticing how the reusable workflows are used, maybe it would have been better to use the following naming instead of the current one:

kyma-project/eventing-tools/.github/workflows/reusable/bump-sec-scanners-config.yaml

but this is only my personal preference, so feel free to ignore.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That might have been a helpful solution. You would still find workflows at the expected place and the extra dir would make clear what this is, especially when we are calling them from other workflow like in this case. Learnings for the future.

VERSION: ${{ needs.gen-version.outputs.VERSION }}
secrets:
BOT_PAT: ${{ secrets.BOT_PAT }}

- name: Generate version number
id: gen-version
run: |
# get script
GET_VERSION=$(mktemp /tmp/get-version-from-branch.XXXXX)
curl -L https://raw.githubusercontent.com/kyma-project/eventing-tools/main/hack/scripts/get-version-from-branch.sh -o "${GET_VERSION}"
chmod +x "${GET_VERSION}"
# get version via script
VERSION=$("${GET_VERSION}")
# push version to output environment file
echo "VERSION=${VERSION}" >> $GITHUB_OUTPUT
run-unit-test:
name: Run Unit Tests
needs: [gen-version, bump-sec-scanners-config]
uses: kyma-project/eventing-tools/.github/workflows/unit-test-reusable.yml@main
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does run-unit-test need bump-sec-scanners-config?

Copy link
Contributor Author

@friedrichwilken friedrichwilken Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not need it, it will just wait to run the unit tests after bump-sec-scanners-config (because of the PR we create in the that part). In general, all the jobs will run consecutively. Only lint and unit-test will run in parallel because they are independent of each other. That flow is controlled by the need instructions.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the only flow that is really needed is the gen-version because only if it get's marked as needs the output (VERSION) gets available in the other jobs.


- name: Check image Tag
env:
VERSION: ${{ steps.gen-version.outputs.VERSION }}
run: ./scripts/check_sec-scanners-config.sh $VERSION
run-golint-cli:
name: Run Golint-cli
needs: [gen-version, bump-sec-scanners-config]
uses: kyma-project/eventing-tools/.github/workflows/lint-reusable.yml@main

create-draft:
name: Create draft release
needs: verify-release
runs-on: ubuntu-latest
env:
VERSION: ${{ needs.verify-release.outputs.VERSION }}
outputs:
release_id: ${{ steps.create-draft.outputs.release_id }}
build-image:
name: Build image
needs: [gen-version, run-unit-test, run-golint-cli]
uses: kyma-project/eventing-tools/.github/workflows/trigger-prow-build-job-reusable.yml@main
with:
VERSION: ${{ needs.gen-version.outputs.VERSION }}
TIMEOUT: 120000 # 20 minutes
INTERVAL: 6000 # 1 minute
CONTEXT: "release-eventing-manager-build"
secrets:
BOT_PAT: ${{ secrets.BOT_PAT }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0
create-draft-release:
name: Create a draft release
needs: [gen-version, run-unit-test, run-golint-cli, build-image]
uses: kyma-project/eventing-tools/.github/workflows/create-draft-release-reusable.yml@main
with:
VERSION: ${{ needs.gen-version.outputs.VERSION }}
secrets:
BOT_PAT: ${{ secrets.BOT_PAT }}
GH_TOKEN: ${{ secrets.GH_TOKEN }}

- name: Create changelog
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/create_changelog.sh $VERSION

- name: Create draft release
id: create-draft
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
RELEASE_ID=$(./scripts/create_draft_release.sh $VERSION)
echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT

- name: Create lightweight tag
run: |
git tag $VERSION
git push origin $VERSION

- name: Verify job status
run: ./scripts/verify-status.sh ${{ github.ref_name }} 600 10 30

- name: Create and upload eventing-manager.yaml and eventing-default-cr.yaml
env:
PULL_BASE_REF: ${{ needs.verify-release.outputs.VERSION }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
IMG: "europe-docker.pkg.dev/kyma-project/prod/eventing-manager:${VERSION}"
MODULE_REGISTRY: "europe-docker.pkg.dev/kyma-project/prod/unsigned"
KUSTOMIZE_VERSION: "v4.5.6"
run: |
./scripts/render_and_upload_manifests.sh
render-and-upload-manifest:
name: Render and upload the manifests
needs: [gen-version, create-draft-release, build-image]
uses: kyma-project/eventing-tools/.github/workflows/render-and-upload-manifests-reusable.yml@main
with:
VERSION: ${{ needs.gen-version.outputs.VERSION }}
CR_FILE: "somedir/cr_file.yaml"
CRD_FILE: "crd_file.yaml"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somedir?

Copy link
Contributor Author

@friedrichwilken friedrichwilken Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh , I forgot to change the names from tests that I have done earlier. Thanks. I fixed it.

But while I fixed this, I noticed that I have done a mistake when I implemented the flow and only tested it with "somedir/cr_file.yaml".

We need a way to rename the default cr, since gh cannot do it or upload it with a different name.

Here is the fix: would you also take a look at this? I will update the commit sha later, once it was merged.
https://github.com/kyma-project/eventing-tools/pull/66/files

secrets:
BOT_PAT: ${{ secrets.BOT_PAT }}

publish-release:
name: Publish release
needs: [verify-release, create-draft]
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Publish release
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: ./scripts/publish_release.sh ${{ needs.create-draft.outputs.release_id }}
if: ${{ !inputs.DRY_RUN }}
needs: [gen-version, render-and-upload-manifest, build-image]
name: Publish the release
uses: kyma-project/eventing-tools/.github/workflows/publish-release-reusable.yml@main
with:
VERSION: ${{ needs.gen-version.outputs.VERSION }}
secrets:
BOT_PAT: ${{ secrets.BOT_PAT }}
57 changes: 57 additions & 0 deletions hack/scripts/create_changelog.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/usr/bin/env bash
Copy link
Contributor

@marcobebway marcobebway Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to move this script to the common tools repo so other repositories may use it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought about this. My intention was to give more flexibility with the changelog (similar to the sec-scanners-config and render-manifest). But you are right, in this case it is not needed at all. I added this PR to do so.


# Error handling.
set -o nounset # treat unset variables as an error and exit immediately.
set -o errexit # exit immediately when a command fails.
set -E # needs to be set if we want the ERR trap
set -o pipefail # prevents errors in a pipeline from being masked

RELEASE_TAG=$1

REPOSITORY=${REPOSITORY:-kyma-project/eventing-manager}
GITHUB_URL=https://api.github.com/repos/${REPOSITORY}
GITHUB_AUTH_HEADER="Authorization: token ${GH_TOKEN}"
CHANGELOG_FILE="CHANGELOG.md"

# The git describe --tag --abbrev=0 command is used to find the most recent tag that is reachable from a commit.
# The --tag option tells git describe to consider any tag found in the refs/tags namespace, enabling matching a lightweight (non-annotated) tag.
PREVIOUS_RELEASE=$(git describe --tags --abbrev=0)

# Generate the changelog in the CHANGELOG.md.
echo "## What has changed" >>${CHANGELOG_FILE}

# Iterate over all commits since the previous release.
git log "${PREVIOUS_RELEASE}"..HEAD --pretty=tformat:"%h" --reverse | while read -r commit; do
# If the author of the commit is not kyma-bot, show append the commit message to the changelog.
friedrichwilken marked this conversation as resolved.
Show resolved Hide resolved
COMMIT_AUTHOR=$(curl -H "${GITHUB_AUTH_HEADER}" -sS "${GITHUB_URL}/commits/${commit}" | jq -r '.author.login')
if [ "${COMMIT_AUTHOR}" != "kyma-bot" ]; then
git show -s "${commit}" --format="* %s by @${COMMIT_AUTHOR}" >>${CHANGELOG_FILE}
fi
done

# Create a new file (with a unique name based on the process ID of the current shell).
NEW_CONTRIB=$$.new
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) $$.new -> $$.authors

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done


# Find unique authors that contribute since the last release, but not before it, and to the NEW_CONTRIB file.
join -v2 \
<(curl -H "${GITHUB_AUTH_HEADER}" -sS "${GITHUB_URL}/compare/$(git rev-list --max-parents=0 HEAD)...${PREVIOUS_RELEASE}" | jq -r '.commits[].author.login' | sort -u) \
<(curl -H "${GITHUB_AUTH_HEADER}" -sS "${GITHUB_URL}/compare/${PREVIOUS_RELEASE}...HEAD" | jq -r '.commits[].author.login' | sort -u) >${NEW_CONTRIB}

# Add new contributors to the 'new contributors' section of the changelog.
if [ -s ${NEW_CONTRIB} ]; then
echo -e "\n## New contributors" >>${CHANGELOG_FILE}
while read -r user; do
REF_PR=$(grep "@${user}" ${CHANGELOG_FILE} | head -1 | grep -o " (#[0-9]\+)" || true)
if [ -n "${REF_PR}" ]; then #reference found
REF_PR=" in ${REF_PR}"
fi
echo "* @${user} made first contribution${REF_PR}" >>${CHANGELOG_FILE}
done <${NEW_CONTRIB}
fi

# Append link to the full-changelog this changelog.
echo -e "\n**Full changelog**: https://github.com/$REPOSITORY/compare/${PREVIOUS_RELEASE}...${RELEASE_TAG}" >>${CHANGELOG_FILE}

# Cleanup the NEW_CONTRIB file.
rm ${NEW_CONTRIB} || echo "cleaned up"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) IMO, no need to echo anything here, because removing the file is an internal housekeeping.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed the whole section. It is just not needed.


42 changes: 42 additions & 0 deletions hack/scripts/render-sec-scanners-config.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#!/usr/bin/env bash

set -e

# This scrpit generates the sec-scanners-config by fetching all relevant images.

TAG=$1
OUTPUT_FILE=${2:-"sec-scanners-config.yaml"}
WEBHOOK_FILE=${3-"config/webhook/kustomization.yaml"}
PUBLISHER_FILE=${4-"config/manager/manager.yaml"}

# Fetch Webhook Image.
echo "fetching webhook image from ${WEBHOOK_FILE}"
WEBHOOK_IMAGE=$(yq eval '.images[0].newName' <"$WEBHOOK_FILE")
WEBHOOK_TAG=$(yq eval '.images[0].newTag' <"$WEBHOOK_FILE")
echo -e "webhook image is ${WEBHOOK_IMAGE}:${WEBHOOK_TAG} \n"
Copy link
Contributor

@marcobebway marcobebway Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(nit) Maybe set the var WEBHOOK_IMAGE to be ${WEBHOOK_IMAGE}:${WEBHOOK_TAG}.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the idea. Done.


# Fetch Publisher Image.
echo "fetching publisher image from ${PUBLISHER_FILE}"
PUBLISHER_IMAGE=$(yq eval '.spec.template.spec.containers[0].env[] | select(.name == "PUBLISHER_IMAGE") | .value' <"${PUBLISHER_FILE}")
echo -e "publisher image is ${PUBLISHER_IMAGE} \n"

# Generating File.
echo -e "generating to ${OUTPUT_FILE} \n"
cat <<EOF | tee "${OUTPUT_FILE}"
# Dont edit this file; it is autogenerated by github action 'Create release'.
# The value for the publisher image are extracted from ${PUBLISHER_FILE}.
# The value for the webhook image are extracted from ${WEBHOOK_FILE}.yaml.
friedrichwilken marked this conversation as resolved.
Show resolved Hide resolved
module-name: eventing
rc-tag: ${TAG}
protecode:
- europe-docker.pkg.dev/kyma-project/prod/eventing-manager:${TAG}
- ${PUBLISHER_IMAGE}
- ${WEBHOOK_IMAGE}:${WEBHOOK_TAG}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After applying this comment, these lines should read:

  - ${PUBLISHER_IMAGE}
  - ${WEBHOOK_IMAGE}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated.

whitesource:
language: golang-mod
subprojects: false
exclude:
- "**/test/**"
- "**/*_test.go"
- "/hack/**"
EOF
10 changes: 10 additions & 0 deletions hack/scripts/render_crd.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
#!/usr/bin/env bash

# standard bash error handling
set -o nounset # treat unset variables as an error and exit immediately.
set -o errexit # exit immediately when a command fails.
set -E # needs to be set if we want the ERR trap
set -o pipefail # prevents errors in a pipeline from being masked

VERSION=$1
IMG="europe-docker.pkg.dev/kyma-project/prod/eventing-manager:${VERSION}" make render-manifest
14 changes: 0 additions & 14 deletions scripts/check_release_tag.sh

This file was deleted.

34 changes: 0 additions & 34 deletions scripts/check_sec-scanners-config.sh

This file was deleted.

56 changes: 0 additions & 56 deletions scripts/create_changelog.sh

This file was deleted.

Loading
Loading