Skip to content

Commit

Permalink
Improve: Consolidate artifact build process for all environments & Ad…
Browse files Browse the repository at this point in the history
…d Workflow for preparing release branch (#4256)

* Prepare CMakeLists.txt for unified builds

- Remove CI_BUILD Cmake variable.
- Convert CI_TAG from ENV variable to CMake variable
- CI_VERSION_PRE_RELEASE can be set in non CI builds

* Prepare build.sh and node Dockerfile for unified builds

- add CI_TAG and CI_VERSION_PRE_RELEASE to build.sh
- add useful ARG with default values to Dockerfile so they can be passed to the build.sh script

* Prepare WIndows for unified builds

- remove CI_BUILD
- convert DCI_TAG from ENV to CMake variable

* Convert TEST BETA and LIVE variable to NETWORK variable for easier workflow matrix usage

* Make docker builds and deploys more modular

- Remove dependency on the workflow name
- COnvert $GITHUB_WORKFLOW to $NETWORK
- Create smaller functions with limited scope
- Create similar deploy functions for docker and github container registries (hub.docker and ghcr)

* Create unified workflow to build TEST BETA and LIVE artifacts

- Remove old workflows
- Create 1 workflow for all environments (Network Matrix)
- Keep current build logic (build scripts still differ per OS)

* Remove hardcoded repo from build and deploy scripts

- skip hub.docker deploys if DOCKER_PASSWORD is not provided
- Create DOCKER_HUB variable which defaults to nanocurrency (backwards compatible)
- Create DOCKER_USER variable which defaults to nanoreleaseteam (backwards compatible)

- create S3_BUCKET_NAME variable that defaults to repo.nano.org if not provided (backwards compatible)
- only use S3_BUILD_DIRECTORY if provided

* Make workflow repository agnostic

Convert nanocurrency/nano-env image to self built ghcr.io/${{ github.repository }} image

* Fix CI_TAG usage

- convert ref to CI_TAG
- use CI_TAG in aws deploys

* Use same login for ghcr and hub.docker

* Fix docker login

- add possibility to specify registry
- use ghcr.io instead of variable for ghcr_image_name

* Fix nano network during build

* Use config variables instead of secrets for non secret variables

- vars.S3_BUCKET_NAME
- vars.S3_BUILD_DIRECTORY
- vars.DOCKER_REGISTRY
- vars.DOCKER_USER

* Fix: Make build process repository agnostic by specifying vars.DOCKER_REGISTRY

* Make tag generation branch agnostic

The goal is to simplify tag generation process and commit the version_pre_release into CMakeLists.txt for each tag.
If a user checks out a specific tag and builds it, the version_pre_release is set correctly.

- remove workflow_dispatch inputs. It operates on the selected branch.
- The cronjob is executed on develop branch only.
- replace ci/actions/dev-build-tag-gen.sh with ci/actions/generate_next_git_tag.sh
- generate_next_git_tag.sh is branch agnostic and operates on  ${{ github.ref }}
- generate_next_git_tag.sh succeeds even if no new tag is generated
- the workflow only executes the build jobs if a new tag was created (if: ${{ needs.prepare_build.outputs.TAG_CREATED == 'true' }})
- generate_next_git_tag.sh uses V${current_version_major}.${current_version_minor}${branch_name} for tags.
- for "develop" branch_name is converted to DB --> (e.g V26.0DB1)
- generate_next_git_tag.sh uses a -c flag that is responsible to update CMakeLists.txt with correct version_pre_release, create and push the tag to origin.

* Remove CI_VERSION_PRE_RELEASE as it has been committed to CMakeLists.txt in the tag used to build the node

* Use the new prepare scripts

- use the new prepare scripts (Linux, MacOS & Windows)
- remove the need the dependency on nano-env:gcc

* Make usage of nano-env:rhel explicit

- Build the nano-env docker image in the current workflow
- Use the locally built image.

* Fix typo in default BUILD_TYPE

- Remove duplicate BUILD_TYPE
- move SANITIZER to ci/build-node.sh ARGs

* Make sure the existing_tags ending is numeric

- fixes a bug when we have 2 similar branches with tags (e.g. some-branch and some-branch-2)

* Remove --global for git config

Prevent modification of global git settings on a developer machine

* Handling release branches differently.

- refactored script by making it more modular
- script expects releases to be made from a branch called `releases/v{Major}`
- add -i flag to indicate wether or not to increment the generated tag
- -i increments version_minor for release branches: tag --> V{Major}.{Minor + increment}
- -i increments version_pre_release for all other non-release branches: tag --> V{Major}.{Minor}{branch_name}{pre_release + increment}
- -o outputs either `version_pre_release` or `version_minor` depending on the branch

Prevent incrementing if no tag exists yet

- make sure V{Major}.0 is created even if the user forgets to set increment to 0

Make tag_created=true explicit

* Prevent output file from being created when -o is not provided

- by redefining local output=$1 it is set to 0 instead of "" and a file called 0 was created

* prevent script exit from grep returning a non-zero exit status if it doesn't find anything

* Add increment input to the workflow and force push if tag is unchanged

- increment is 1 by default
- if increment is 0, an existing tag will be updated (origin push -f).

fixup! Add increment input to the workflow

* Allow tag creation without file changes

- the first tag of a releases/ branch creates no changes
- a tag run with increment=0 creates no changes
in both cases we want to create the tag (again)

* feat: implement workflow to prepare for a new major software release

- Checks out the repository based on github.ref.
- Extracts and sets current major and minor versions from CMakeLists.txt of github.ref.
- Fetches the default branch name and sets it as an output.
- Checks for existence of a corresponding release branch. If present, the workflow aborts, if not, it continues and sets the new release branch name.
- Increments the major version in CMakeLists.txt on the default branch, commits this change, and pushes it back.
- Checks out a new release branch, sets the pre-release version to 0 in CMakeLists.txt, commits this change, and pushes the new branch to the repository.

fix: Configure git and fix output variables

Change the order of steps to make the flow more natural

Fix curl DEFAULT_BRANCH

fix

fix

* Rename workflow to something more meaningful

* Remove increment input from workflow

- Autodetect if we need to increment the version or not on the release branch
Expected behaviour :
- Running the workflow multiple times without a new commit will rebuild and republish the same artifacts
- Running the workflow after a new commit was made to the releases branch will increment version_minor by 1
  - the version_minor is updated in CMakeLists.txt and committed to the release branch
  - a new tag is created and pushed with the incremented version_minor
- Make logic for both release and other branches more obvious

 --> TODO?: instead of force pushing, we could remove -f option if branch or tag already exist

* Retry "brew update" up to 5 times with 15s interval

* Apply suggestions from PR review (1/2)

* Apply suggestions from PR review (2/2)

Convert if/else into case..in

---------

Co-authored-by: gr0vity-dev <[email protected]>
  • Loading branch information
gr0vity-dev and gr0vity-dev authored Jul 20, 2023
1 parent c7aa465 commit 4756801
Show file tree
Hide file tree
Showing 25 changed files with 794 additions and 822 deletions.
217 changes: 217 additions & 0 deletions .github/workflows/artifacts_build_deploy.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,217 @@
name: Build & Deploy
on:
schedule:
- cron: "0 0 * * 3,6"
workflow_dispatch:

env:
artifact: 1

jobs:
prepare_build:
runs-on: ubuntu-22.04
outputs:
CI_TAG: ${{ steps.tag_set.outputs.CI_TAG }}
TAG_CREATED: ${{ steps.tag_set.outputs.tag_created }}
steps:
- uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 #v3.1.0
with:
ref: ${{ github.ref }}
- name: Set the tag and version
id: tag_set
run: |
output_var_file="variable_list.txt"
ci/actions/generate_next_git_tag.sh -c -o "${output_var_file}"
CI_TAG=$(grep 'build_tag' ${output_var_file} | cut -d= -f2)
echo "CI_TAG=${CI_TAG}" >> $GITHUB_OUTPUT
TAG_CREATED=$(grep 'tag_created' ${output_var_file} | cut -d= -f2)
echo "TAG_CREATED=${TAG_CREATED}" >> $GITHUB_OUTPUT
env:
GITHUB_ACTOR: ${{ github.actor }}


osx_job:
needs: prepare_build
if: ${{ needs.prepare_build.outputs.TAG_CREATED == 'true' }}
runs-on: macOS-12
timeout-minutes: 90
strategy:
matrix:
network: ["TEST", "BETA", "LIVE"]
steps:

- uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 #v3.1.0
with:
submodules: "recursive"
ref: ${{ needs.prepare_build.outputs.CI_TAG }}
repository: ${{ github.repository }}
- name: Prepare
run: ci/prepare/macos/prepare.sh
- name: Build Artifact
run: ci/build-deploy.sh "/tmp/qt/lib/cmake/Qt5";
env:
NETWORK: ${{ matrix.network }}
CI_TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
- name: Deploy Artifact
run: ci/actions/deploy.sh
env:
NETWORK: ${{ matrix.network }}
TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
S3_BUCKET_NAME: ${{ vars.S3_BUCKET_NAME }}
S3_BUILD_DIRECTORY: ${{ vars.S3_BUILD_DIRECTORY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-2


linux_job:
needs: prepare_build
if: ${{ needs.prepare_build.outputs.TAG_CREATED == 'true' }}
runs-on: ubuntu-22.04
timeout-minutes: 90
strategy:
matrix:
network: ["TEST", "BETA", "LIVE"]
steps:
- uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 #v3.1.0
with:
submodules: "recursive"
ref: ${{ needs.prepare_build.outputs.CI_TAG }}
repository: ${{ github.repository }}
- name: Prepare
run: sudo -E ci/prepare/linux/prepare.sh
- name: Build Artifact
run: ci/build-deploy.sh "/usr/lib/x86_64-linux-gnu/cmake/Qt5"
env:
NETWORK: ${{ matrix.network }} \
CI_TAG: ${{ needs.prepare_build.outputs.CI_TAG }} \

- name: Deploy Artifact
run: ci/actions/deploy.sh
env:
NETWORK: ${{ matrix.network }}
TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
S3_BUCKET_NAME: ${{ vars.S3_BUCKET_NAME }}
S3_BUILD_DIRECTORY: ${{ vars.S3_BUILD_DIRECTORY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-2


linux_rpm_job:
needs: prepare_build
if: ${{ needs.prepare_build.outputs.TAG_CREATED == 'true' }}
runs-on: ubuntu-22.04
timeout-minutes: 90
strategy:
matrix:
network: ["BETA", "LIVE"] #No path to build TEST exists ci/build-rhel.sh
steps:
- uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 #v3.1.0
with:
submodules: "recursive"
ref: "develop" #build-rhel.sh needs develop branch and then sets the tag
repository: ${{ github.repository }}
- name: Build local/nano-env:rhel
run: ci/actions/linux/install_deps.sh
env:
COMPILER: rhel
DOCKER_REGISTRY: local
- name: RockyLinux 8 Base
run: ci/build-docker-image.sh docker/ci/Dockerfile-rhel local/nano-env:rhel
- name: Build Artifact
run: |
mkdir -p ${GITHUB_WORKSPACE}/artifacts
docker run -v ${GITHUB_WORKSPACE}:/workspace -v ${GITHUB_WORKSPACE}/artifacts:/root/rpmbuild \
local/nano-env:rhel /bin/bash -c " \
NETWORK=${{ matrix.network }} \
TAG=${{ needs.prepare_build.outputs.CI_TAG }} \
REPO_TO_BUILD=${{ github.repository }} \
RPM_RELEASE=1 \
ci/build-rhel.sh"
- name: Deploy Artifacts
run: ci/actions/deploy.sh
env:
LINUX_RPM: 1
NETWORK: ${{ matrix.network }}
# TAG: ${{ needs.prepare_build.outputs.CI_TAG }} # (not used in the deploy script if LINUX_RPM==1 )
S3_BUCKET_NAME: ${{ vars.S3_BUCKET_NAME }}
S3_BUILD_DIRECTORY: ${{ vars.S3_BUILD_DIRECTORY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-2


linux_docker_job:
needs: prepare_build
if: ${{ needs.prepare_build.outputs.TAG_CREATED == 'true' }}
runs-on: ubuntu-22.04
timeout-minutes: 90
strategy:
matrix:
network: ["TEST", "BETA", "LIVE"]
steps:
- uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 #v3.1.0
with:
submodules: "recursive"
ref: ${{ needs.prepare_build.outputs.CI_TAG }}
repository: ${{ github.repository }}
- name: Build Docker
run: ci/actions/linux/docker-build.sh
env:
NETWORK: ${{ matrix.network }}
CI_TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
DOCKER_REGISTRY: ${{ vars.DOCKER_REGISTRY }}
- name: Check if secrets.DOCKER_PASSWORD exists
run: echo "DOCKER_PASSWORD_EXISTS=${{ secrets.DOCKER_PASSWORD != '' }}" >> $GITHUB_ENV
- name: Deploy Docker Hub
if: env.DOCKER_PASSWORD_EXISTS == 'true'
run: ci/actions/linux/docker-deploy.sh
env:
CI_TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
NETWORK: ${{ matrix.network }}
DOCKER_REGISTRY: ${{ vars.DOCKER_REGISTRY }}
DOCKER_USER: ${{ vars.DOCKER_USER }}
DOCKER_PASSWORD: ${{ secrets.DOCKER_PASSWORD }}
- name: Deploy Docker (ghcr.io)
run: ci/actions/linux/ghcr-deploy.sh
env:
CI_TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
NETWORK: ${{ matrix.network }}
DOCKER_REGISTRY: ghcr.io
DOCKER_USER: ${{ github.repository_owner }}
DOCKER_PASSWORD: ${{ secrets.GITHUB_TOKEN }}


windows_job:
needs: prepare_build
if: ${{ needs.prepare_build.outputs.TAG_CREATED == 'true' }}
runs-on: windows-latest
timeout-minutes: 90
strategy:
matrix:
network: ["TEST", "BETA", "LIVE"]
steps:
- uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 #v3.1.0
with:
submodules: "recursive"
ref: ${{ needs.prepare_build.outputs.CI_TAG }}
repository: ${{ github.repository }}
- name: Prepare
run: ci/prepare/windows/prepare.ps1
- name: Build Artifact
run: ci/actions/windows/build.ps1
env:
CSC_LINK: ${{ secrets.CSC_LINK }}
CSC_KEY_PASSWORD: ${{ secrets.CSC_KEY_PASSWORD }}
- name: Deploy Artifact
run: ci/actions/windows/deploy.ps1
env:
NETWORK: ${{ matrix.network }}
TAG: ${{ needs.prepare_build.outputs.CI_TAG }}
S3_BUCKET_NAME: ${{ vars.S3_BUCKET_NAME }}
S3_BUILD_DIRECTORY: ${{ vars.S3_BUILD_DIRECTORY }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_DEFAULT_REGION: us-east-2
Loading

0 comments on commit 4756801

Please sign in to comment.