Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DPC-4496] use upload/download artifacts for rails docker builds #2429

Merged
merged 53 commits into from
Feb 7, 2025
Merged

[DPC-4496] use upload/download artifacts for rails docker builds #2429

Changes from all commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
f7410e1
submit new job to build java apps and push to ecr
lukey-luke Jan 31, 2025
960c083
push trigger to manually verify job
lukey-luke Feb 1, 2025
a0a17e2
ls to check why cleanup script is failing
lukey-luke Feb 1, 2025
214f2f3
remove calls to cleanup script due to rate limiting
lukey-luke Feb 1, 2025
f0b8615
use upload artifacts instead
lukey-luke Feb 3, 2025
1101685
fix env vars
lukey-luke Feb 3, 2025
b12b60d
fix path for artifact
lukey-luke Feb 3, 2025
ecb59ed
try including runner.temp for gzip filepath??
lukey-luke Feb 4, 2025
041c462
fix tar name
lukey-luke Feb 4, 2025
94f8120
manually write out repo name
lukey-luke Feb 4, 2025
a9a10a7
write it out manual for upload too??
lukey-luke Feb 4, 2025
ff3f99d
push trigger to manually verify job
lukey-luke Feb 1, 2025
37e50a6
use upload artifacts instead
lukey-luke Feb 3, 2025
22c73cb
fix env vars
lukey-luke Feb 3, 2025
d6f244b
fix path for artifact
lukey-luke Feb 3, 2025
76d9679
try including runner.temp for gzip filepath??
lukey-luke Feb 4, 2025
de17f64
fix tar name
lukey-luke Feb 4, 2025
93e5f73
manually write out repo name
lukey-luke Feb 4, 2025
4a24853
write it out manual for upload too??
lukey-luke Feb 4, 2025
b0e7068
Merge remote-tracking branch 'origin/ls/task-4496-artifacts-for-build…
lukey-luke Feb 5, 2025
3bf563a
fix tar path for docker load
lukey-luke Feb 5, 2025
3f9c0d3
create a matrix in order to have reusable GHA code across portals
lukey-luke Feb 5, 2025
abd2a4e
create a temporary hardcoded tag while investigating disk usage
lukey-luke Feb 6, 2025
bbbdf53
additional commands to investigate disk usage
lukey-luke Feb 6, 2025
e4cbe5c
clean up branch
lukey-luke Feb 6, 2025
9b684bc
push trigger to manually verify job
lukey-luke Feb 1, 2025
8bb7255
use upload artifacts instead
lukey-luke Feb 3, 2025
2b76c97
fix env vars
lukey-luke Feb 3, 2025
d70bb9c
fix path for artifact
lukey-luke Feb 3, 2025
a0de5f3
try including runner.temp for gzip filepath??
lukey-luke Feb 4, 2025
2366322
fix tar name
lukey-luke Feb 4, 2025
7496852
manually write out repo name
lukey-luke Feb 4, 2025
1fe0a32
write it out manual for upload too??
lukey-luke Feb 4, 2025
c602dba
submit new job to build java apps and push to ecr
lukey-luke Jan 31, 2025
6d5357c
ls to check why cleanup script is failing
lukey-luke Feb 1, 2025
d178ca8
remove calls to cleanup script due to rate limiting
lukey-luke Feb 1, 2025
3f40205
use upload artifacts instead
lukey-luke Feb 3, 2025
bba8bff
fix env vars
lukey-luke Feb 3, 2025
2535639
fix path for artifact
lukey-luke Feb 3, 2025
27be142
try including runner.temp for gzip filepath??
lukey-luke Feb 4, 2025
ba1f01c
fix tar name
lukey-luke Feb 4, 2025
b3511eb
manually write out repo name
lukey-luke Feb 4, 2025
1c5663e
write it out manual for upload too??
lukey-luke Feb 4, 2025
1ec282f
fix tar path for docker load
lukey-luke Feb 5, 2025
d2c643a
create a matrix in order to have reusable GHA code across portals
lukey-luke Feb 5, 2025
2076fd7
create a temporary hardcoded tag while investigating disk usage
lukey-luke Feb 6, 2025
f74b604
additional commands to investigate disk usage
lukey-luke Feb 6, 2025
6248f7b
clean up branch
lukey-luke Feb 6, 2025
79d611a
Merge remote-tracking branch 'origin/ls/task-4496-artifacts-for-build…
lukey-luke Feb 6, 2025
f1ba69e
pull in java changes from main
lukey-luke Feb 6, 2025
be5630c
update make command
lukey-luke Feb 6, 2025
ca27eef
rename occurrences of repository w/ ecr_repository to be more explicit
lukey-luke Feb 6, 2025
91b03e1
set retention period for artifacts
lukey-luke Feb 6, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 58 additions & 99 deletions .github/workflows/docker_build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ name: Docker Build
on:
workflow_dispatch:


permissions:
id-token: write
contents: read
Expand All @@ -15,8 +14,20 @@ env:
ENV: "github-ci"

jobs:
docker_build_rails_web_portal:
docker_build_rails_apps:
runs-on: self-hosted
strategy:
matrix:
ecr_repository: [ web-portal, web-admin, web ]
include:
# note this is confusing, but make ci-web-portal points to dpc-web-portal-test.sh which runs
# docker compose -p ... dpc_web
- ecr_repository: web-portal
make_command: make ci-portal
- ecr_repository: web-admin
make_command: make ci-admin-portal
- ecr_repository: web
make_command: make ci-web-portal
steps:
- name: Install python3
run: sudo dnf install python3
Expand All @@ -31,107 +42,55 @@ jobs:
sudo chown root:root /usr/local/lib/docker/cli-plugins/docker-compose
sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose

# TODO replace w/ make ci-portals-v1 when disk space issue resolved
- name: Build portal
- name: Build specified app
run: ${{ matrix.make_command }}

- name: gzip the image
run: docker save dpc-${{ matrix.ecr_repository }}:latest | gzip > ${{ runner.temp }}/dpc_${{ matrix.ecr_repository }}_latest.tar.gz
- name: upload tar artifact
uses: actions/upload-artifact@v4
with:
name: dpc-${{ matrix.ecr_repository }}
path: ${{ runner.temp }}/dpc_${{ matrix.ecr_repository }}_latest.tar.gz
retention-days: 1

docker_push_rails_apps:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't all apps be pushed together?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, breaking up the java build to use upload/download artifacts and then having a single job waiting for all should achieve this. I'll work on a follow-up PR for this

runs-on: self-hosted
strategy:
matrix:
ecr_repository: [ web-portal, web-admin, web ]
env:
ECR_REPOSITORY: ${{ matrix.ecr_repository }}
needs: docker_build_rails_apps
steps:
- name: Download artifact
uses: actions/download-artifact@v4
with:
name: dpc-${{ matrix.ecr_repository }}
path: ${{ runner.temp }}
- name: Load docker image from artifact download
run: |
make ci-portal
docker load --input ${{ runner.temp }}/dpc_${{ matrix.ecr_repository }}_latest.tar.gz
docker image ls -a

- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-region: ${{ vars.AWS_REGION }}
role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/dpc-dev-github-actions

- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v2

- name: Push to ECR
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: web-portal
IMAGE_TAG: ${{ github.sha }}
run: |
docker tag dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:latest
docker tag dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
# TODO add additional steps to push dpc-web-admin and dpc-web, see docker_build_all_portals below

# docker_build_all_portals:
# runs-on: self-hosted
# steps:
# - name: Install python3
# run: sudo dnf install python3
#
# - name: "Checkout code"
# uses: actions/checkout@v4
# with:
# ref: ${{ github.ref_name }}
#
# - name: Install docker compose manually
# run: |
# sudo mkdir -p /usr/local/lib/docker/cli-plugins
# sudo curl -SL https://github.com/docker/compose/releases/download/v2.32.4/docker-compose-linux-x86_64 -o /usr/local/lib/docker/cli-plugins/docker-compose
# sudo chown root:root /usr/local/lib/docker/cli-plugins/docker-compose
# sudo chmod +x /usr/local/lib/docker/cli-plugins/docker-compose
#
# - name: Assert Ownership
# run: sudo chmod -R 777 .
# - name: Cleanup Runner
# run: ./scripts/cleanup-docker.sh
#
# - name: Configure AWS Credentials
# uses: aws-actions/configure-aws-credentials@v4
# with:
# aws-region: ${{ vars.AWS_REGION }}
# role-to-assume: arn:aws:iam::${{ secrets.ACCOUNT_ID }}:role/delegatedadmin/developer/dpc-dev-github-actions
#
# - name: Login to Amazon ECR
# id: login-ecr
# uses: aws-actions/amazon-ecr-login@v2
#
# - name: "Set up Ansible"
# run: |
# sudo dnf -y install python3 python3-pip
# pip install ansible
#
# - name: Build portals
# run: |
# make ci-portals-v1
#
# - name: Push Rails Web Portal to ECR
# env:
# REGISTRY: ${{ steps.login-ecr.outputs.registry }}
# REPOSITORY: web-portal
# IMAGE_TAG: hardcodedstringfornow
# run: |
# docker tag dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
# docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
#
# - name: Push Rails Admin Portal to ECR
# env:
# REGISTRY: ${{ steps.login-ecr.outputs.registry }}
# REPOSITORY: web-admin
# IMAGE_TAG: hardcodedstringfornow
# run: |
# docker tag dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
# docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
#
# - name: Push Web to ECR
# env:
# REGISTRY: ${{ steps.login-ecr.outputs.registry }}
# REPOSITORY: web
# IMAGE_TAG: hardcodedstringfornow
# run: |
# docker tag dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
# docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
#
# - name: echo hello
# run: echo "pushed portal images"
#
# - name: Cleanup at the end too??
# if: ${{ always() }}
# run: ./scripts/cleanup-docker.sh
docker tag dpc-$ECR_REPOSITORY:latest $REGISTRY/dpc-$ECR_REPOSITORY:latest
docker tag dpc-$ECR_REPOSITORY:latest $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG


docker_build_java:
runs-on: self-hosted
Expand Down Expand Up @@ -209,35 +168,35 @@ jobs:
- name: ECR (1 of 4) - Push API
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: api
ECR_REPOSITORY: api
IMAGE_TAG: ${{ github.sha }}
run: |
docker tag $REGISTRY/dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker tag $REGISTRY/dpc-$ECR_REPOSITORY:latest $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG

- name: ECR (2 of 4) - Push Attribution
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: attribution
ECR_REPOSITORY: attribution
IMAGE_TAG: ${{ github.sha }}
run: |
docker tag $REGISTRY/dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker tag $REGISTRY/dpc-$ECR_REPOSITORY:latest $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG

- name: ECR (3 of 4) - Push Aggregation
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: aggregation
ECR_REPOSITORY: aggregation
IMAGE_TAG: ${{ github.sha }}
run: |
docker tag $REGISTRY/dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker tag $REGISTRY/dpc-$ECR_REPOSITORY:latest $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG

- name: ECR (4 of 4) - Push Consent
env:
REGISTRY: ${{ steps.login-ecr.outputs.registry }}
REPOSITORY: consent
ECR_REPOSITORY: consent
IMAGE_TAG: ${{ github.sha }}
run: |
docker tag $REGISTRY/dpc-$REPOSITORY:latest $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$REPOSITORY:$IMAGE_TAG
docker tag $REGISTRY/dpc-$ECR_REPOSITORY:latest $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG
docker push $REGISTRY/dpc-$ECR_REPOSITORY:$IMAGE_TAG