Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch AWS staging hub to GitHub Actions #754

Merged
merged 24 commits into from
Oct 6, 2020
Merged

Switch AWS staging hub to GitHub Actions #754

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
14cf4ab
Basic GitHub Action
salvis2 Sep 24, 2020
3ba3954
move hubploy image to env var
salvis2 Sep 24, 2020
7294c88
Typos / formatting of step names
salvis2 Sep 24, 2020
ccc5daf
Make helm install command many lines, add in other helm repos
salvis2 Sep 24, 2020
c211073
Clean up multi-line yaml
salvis2 Sep 24, 2020
cede57e
Add EKS IP Whitelisting
salvis2 Sep 24, 2020
57528a6
Remove AWS CI actions from CircleCI
salvis2 Sep 24, 2020
d8dc6c2
Move and rename aws-specific github action
salvis2 Sep 30, 2020
2bc87d2
Move image building to its own action
salvis2 Sep 30, 2020
9c96d2e
Change action name to Deploy AWS hub
salvis2 Sep 30, 2020
71c4fed
Make more specific name to Build AWS Image
salvis2 Sep 30, 2020
8a508b3
Add paths for filtering action triggers
salvis2 Sep 30, 2020
7ab78f7
Rename build action file for clarity
salvis2 Oct 1, 2020
509abac
Remove check-registry flag
salvis2 Oct 1, 2020
927f18c
Test changes to this PR branch on my fork
salvis2 Oct 1, 2020
e35fa82
Allow the deployment of staging on this branch
salvis2 Oct 1, 2020
dbab31b
Correct workflow files so that Actions run correctly
salvis2 Oct 1, 2020
c51b326
Switch to checkoutv2
salvis2 Oct 2, 2020
4f003f7
use if always to make sure we always revert the IP list
salvis2 Oct 2, 2020
02c15e6
Remove hubploy build blocks that were commented out
salvis2 Oct 2, 2020
b04302a
Add workflow file as a path to trigger action on PRs
salvis2 Oct 2, 2020
f556455
Be more specific with acceptable paths on deploy command
salvis2 Oct 2, 2020
a6b75e8
Add in the hyphen to make the Helm commands a step
salvis2 Oct 2, 2020
272c4ad
Remove workflow triggers on the switch-to-github-actions branch
salvis2 Oct 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 35 additions & 35 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,13 +53,13 @@ jobs:
command: |
hubploy build gcp-uscentral1b --commit-range ${COMMIT_RANGE}

- run:
name: Test building aws-uswest2.pangeo.io image if needed
when: always
command: |
hubploy build icesat2 --commit-range ${COMMIT_RANGE}
environment:
PYTHONIOENCODING: utf-8
#- run:
# name: Test building aws-uswest2.pangeo.io image if needed
# when: always
# command: |
# hubploy build icesat2 --commit-range ${COMMIT_RANGE}
# environment:
# PYTHONIOENCODING: utf-8

# This is currently disabled becauses it always runs and because the Azure
# container registry is not open to annonymous read access. This can be changed
Expand Down Expand Up @@ -142,13 +142,13 @@ jobs:
command: |
hubploy build gcp-uscentral1b --check-registry --push

- run:
name: Build aws-uswest2.pangeo.io image if needed
when: always
command: |
hubploy build icesat2 --check-registry --push
environment:
PYTHONIOENCODING: utf-8
#- run:
# name: Build aws-uswest2.pangeo.io image if needed
# when: always
# command: |
# hubploy build icesat2 --check-registry --push
# environment:
# PYTHONIOENCODING: utf-8

- run:
name: Build ooi.pangeo.io image if needed
Expand Down Expand Up @@ -202,28 +202,28 @@ jobs:

# NOTE: should move the dynamic IP into hubploy where credentials and awscli version already dealt with
# sleep 2min for now, but better to poll for readiness https://docs.aws.amazon.com/eks/latest/userguide/cluster-endpoint.html
- run:
name: Add Runner IP to EKS Kubernetes API Whitelist
when: always
command: |
export AWS_SHARED_CREDENTIALS_FILE=./deployments/icesat2/secrets/aws-config.txt
RUNNERIP=`curl --silent https://checkip.amazonaws.com`
aws --version
aws eks update-cluster-config --region us-west-2 --name pangeo --resources-vpc-config publicAccessCidrs=${RUNNERIP}/32 > /dev/null
sleep 120
- run:
name: Deploy aws-uswest2.pangeo.io
when: always
no_output_timeout: 1200
command: |
hubploy deploy icesat2 pangeo-deploy ${CIRCLE_BRANCH} --timeout 1200s --cleanup-on-fail
#- run:
# name: Add Runner IP to EKS Kubernetes API Whitelist
# when: always
# command: |
# export AWS_SHARED_CREDENTIALS_FILE=./deployments/icesat2/secrets/aws-config.txt
# RUNNERIP=`curl --silent https://checkip.amazonaws.com`
# aws --version
# aws eks update-cluster-config --region us-west-2 --name pangeo --resources-vpc-config publicAccessCidrs=${RUNNERIP}/32 > /dev/null
# sleep 120
#- run:
# name: Deploy aws-uswest2.pangeo.io
# when: always
# no_output_timeout: 1200
# command: |
# hubploy deploy icesat2 pangeo-deploy ${CIRCLE_BRANCH} --timeout 1200s --cleanup-on-fail

- run:
name: Revert to Original EKS IP Whitelist
when: always
command: |
export AWS_SHARED_CREDENTIALS_FILE=./deployments/icesat2/secrets/aws-config.txt
aws eks update-cluster-config --region us-west-2 --name pangeo --resources-vpc-config publicAccessCidrs=${AWS_IP_WHITELIST} > /dev/null
#- run:
# name: Revert to Original EKS IP Whitelist
# when: always
# command: |
# export AWS_SHARED_CREDENTIALS_FILE=./deployments/icesat2/secrets/aws-config.txt
# aws eks update-cluster-config --region us-west-2 --name pangeo --resources-vpc-config publicAccessCidrs=${AWS_IP_WHITELIST} > /dev/null

- run:
name: Deploy ooi.pangeo.io
Expand Down
75 changes: 75 additions & 0 deletions .github/workflows/build-aws-hub.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
name: Deploy AWS Hub
on:
push:
branches:
- staging
#- prod
paths:
- 'deployments/icesat2/**'
- 'pangeo-deploy/**'
paths_ignore:
- 'deployments/icesat2/image/binder/*'

env:
HUBPLOY_IMAGE: docker://yuvipanda/hubploy:20200826083951674280

jobs:
deploy_aws:
name: Deploy AWS Hub
# This job runs on Linux
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
salvis2 marked this conversation as resolved.
Show resolved Hide resolved
- uses: $HUBPLOY_IMAGE
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i wonder what it would take to package hubploy as an action to use the standard pattern like -uses: hubploy/hubploy@v1 (https://docs.github.com/en/free-pro-team@latest/actions/creating-actions/creating-a-docker-container-action).

Copy link
Member Author

@salvis2 salvis2 Oct 2, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yuvipanda , do you have interest in this? I'm wondering if that would make it easier to use / manage hubploy for people. Currently, some pieces of the action feel a bit hacky, see things like this from IP whitelisting:

with:
  entrypoint: /bin/bash
  args: >
    -c "code &&
    more code &&
    end of code"

name: Unlock git-crypt Secrets
env:
GIT_CRYPT_KEY: ${{ secrets.GIT_CRYPT_KEY }}
with:
entrypoint: /bin/bash
args: -c "echo ${GIT_CRYPT_KEY} | base64 -d | git crypt unlock - && git crypt status"
#- uses: $HUBPLOY_IMAGE
# name: Build & Push AWS Image if Needed
# with:
# args: -c "build icesat2 --check-registry --push"
- uses: $HUBPLOY_IMAGE
name: Setup Helm
with:
entrypoint: /bin/bash
args: >
-c "helm init --client-only &&
helm repo add jupyterhub https://jupyterhub.github.io/helm-chart/ &&
helm repo add dask https://helm.dask.org/ &&
helm repo add dask-gateway https://dask.org/dask-gateway-helm-repo/ &&
helm repo add stable https://kubernetes-charts.storage.googleapis.com &&
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx &&
helm repo add prometheus-operator https://kubernetes-charts.storage.googleapis.com &&
helm repo update"
- uses: $HUBPLOY_IMAGE
name: Add Runner IP to EKS Kubernetes API Whitelist
with:
entrypoint: /bin/bash
args: >
-c "export AWS_SHARED_CREDENTIALS_FILE=./deployments/icesat2/secrets/aws-config.txt &&
RUNNERIP=`curl --silent https://checkip.amazonaws.com` &&
aws --version &&
aws eks update-cluster-config --region us-west-2 --name pangeo --resources-vpc-config publicAccessCidrs=${RUNNERIP}/32 > /dev/null &&
sleep 120"
- uses: $HUBPLOY_IMAGE
name: Deploy AWS Staging Hub
if: github.ref == 'refs/heads/staging'
with:
args: deploy icesat2 pangeo-deploy staging --timeout 1200s --cleanup-on-fail
- uses: $HUBPLOY_IMAGE
name: Revert to Original EKS IP Whitelist
salvis2 marked this conversation as resolved.
Show resolved Hide resolved
env:
AWS_IP_WHITELIST: ${{ secrets.AWS_IP_WHITELIST }}
with:
entrypoint: /bin/bash
args: >
-c "export AWS_SHARED_CREDENTIALS_FILE=./deployments/icesat2/secrets/aws-config.txt &&
aws eks update-cluster-config --region us-west-2 --name pangeo --resources-vpc-config publicAccessCidrs=${AWS_IP_WHITELIST} > /dev/null"
#- uses: $HUBPLOY_IMAGE
# name: Deploy AWS Production Hub
# if: github.ref == 'refs/heads/prod'
# with:
# args: deploy icesat2 pangeo-deploy prod --timeout 1200s --cleanup-on-fail
35 changes: 35 additions & 0 deletions .github/workflows/build-aws-image.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Build AWS Image
on:
push:
branches:
- staging
#- prod
paths:
- 'deployments/icesat2/image/binder/*'
pull_request:
branches:
- staging
#- prod
paths:
- 'deployments/icesat2/image/binder/*'

env:
HUBPLOY_IMAGE: docker://yuvipanda/hubploy:20200826083951674280

jobs:
build_aws_image:
name: Build AWS Image
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v1
- uses: $HUBPLOY_IMAGE
name: Unlock git-crypt Secrets
env:
GIT_CRYPT_KEY: ${{ secrets.GIT_CRYPT_KEY }}
with:
entrypoint: /bin/bash
args: -c "echo ${GIT_CRYPT_KEY} | base64 -d | git crypt unlock - && git crypt status"
- uses: $HUBPLOY_IMAGE
name: Build & Push AWS Image if Needed
with:
args: -c "build icesat2 --push"