Skip to content

Commit

Permalink
fix: ensure only the latest stac documents are synced into s3 (#300)
Browse files Browse the repository at this point in the history
**Motivation**
When jobs are rerun STAC documents can be synced into s3 from a old pull
request

**Modification**
allows workflows to be run without a commit merged to ensure stac is
upto date
Force checkout master when syncing stac so only the latest versions are
synced.
Also updates some GitHub Action versions and the Pull Request template
to match the imagery repo.
  • Loading branch information
amfage authored Jun 24, 2024
1 parent 58e5e61 commit 1e21ade
Show file tree
Hide file tree
Showing 2 changed files with 47 additions and 5 deletions.
13 changes: 13 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
### Motivation

<!-- TODO: Say why you made your changes. -->

### Modifications

<!-- TODO: Say what changes you made. -->

<!-- TODO: Attach screenshots if you changed the UI. -->

### Verification

<!-- TODO: Say how you tested your changes. -->
39 changes: 34 additions & 5 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -1,74 +1,93 @@
name: Publish
on: [push]
on:
push:
workflow_dispatch:

jobs:
main:
name: Validate STAC
runs-on: ubuntu-latest
steps:
- uses: linz/action-typescript@dee99184c4305aea6c380a52db9b2d7abaaa3e78 # v3
- uses: linz/action-typescript@9bf69b0f313b3525d3ba3116f26b1aff7eb7a6c0 # v3
with:
node-version: 20.x
# FIXME: catalog.json is not pushed to the repository (temporary solution)

- name: Create STAC Catalog
uses: docker://ghcr.io/linz/argo-tasks:v4
with:
args: stac-catalog --output stac/catalog.json --template template/catalog.json /github/workspace/stac/

- name: Validate STAC Catalog
uses: docker://ghcr.io/linz/argo-tasks:v4
with:
args: stac-validate /github/workspace/stac/catalog.json

- name: Validate STAC Collections
run: |
# Enable double star operator
shopt -s globstar
docker run -v "$PWD:$PWD" ghcr.io/linz/argo-tasks:v4 stac-validate "$PWD"/stac/**/collection.json
- name: Download actionlint
run: docker build --tag actionlint - < .github/workflows/actionlint.dockerfile

- name: Run actionlint to check workflow files
run: docker run --volume="${PWD}:/repo" --workdir=/repo actionlint -color

publish-odr:
name: Publish ODR
runs-on: ubuntu-latest
needs: main
if: ${{ github.ref == 'refs/heads/master' }}

environment:
name: prod

permissions:
id-token: write
contents: read

steps:
- uses: linz/action-typescript@dee99184c4305aea6c380a52db9b2d7abaaa3e78 # v3
- uses: linz/action-typescript@9bf69b0f313b3525d3ba3116f26b1aff7eb7a6c0 # v3
with:
node-version: 20.x

- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
with:
fetch-depth: 0

- name: Use Node.js 18.x
uses: actions/setup-node@8f152de45cc393bb48ce5d89d36b731f54556e65 # v4.0.0
with:
node-version: '18.x'

- name: Setup kubectl
uses: azure/setup-kubectl@901a10e89ea615cf61f57ac05cecdf23e7de06d8 # v3
uses: azure/setup-kubectl@3e0aec4d80787158d308d7b364cb1b702e7feb7f # v4.0.0
with:
version: 'latest'

- name: AWS Configure
uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4
with:
aws-region: ap-southeast-2
mask-aws-account-id: true
role-to-assume: ${{ secrets.AWS_EKS_CI_ROLE }}

- name: Login to EKS
run: |
aws eks update-kubeconfig --name Workflows --region ap-southeast-2
- name: Check EKS connection
run: |
kubectl get nodes
- name: Install Argo
run: |
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.4.11/argo-linux-amd64.gz
gunzip argo-linux-amd64.gz
chmod +x argo-linux-amd64
./argo-linux-amd64 version
- name: Submit Added/Changed Parameter Files
id: modified-files
if: ${{ !contains(github.event.head_commit.message, '[skip-argo]')}}
Expand All @@ -79,36 +98,46 @@ jobs:
for file in "${modified_parameter_files[@]}"; do
./argo-linux-amd64 submit --wait --from wftmpl/copy -n argo -f "$file" -p aws_role_config_path="s3://linz-bucket-config/config-write.open-data-registry.json" -p exclude="collection.json$" --generate-name "publish-odr-file-copy-"
done
sync-stac:
name: Sync STAC files
runs-on: ubuntu-latest
concurrency: publish-${{ github.ref }}
needs: publish-odr
if: ${{ github.ref == 'refs/heads/master' }}

environment:
name: prod

permissions:
id-token: write
contents: read

steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
with:
ref: master # Ensure only the latest STAC documents are synced into S3

- name: AWS Configure
uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4
with:
aws-region: ap-southeast-2
mask-aws-account-id: true
role-to-assume: ${{ secrets.AWS_ODR_CI_ROLE }}

- name: AWS Configure ODR
uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4
with:
aws-region: ap-southeast-2
mask-aws-account-id: true
role-to-assume: ${{ secrets.AWS_ODR_DATA_MANAGER_ROLE }}
role-chaining: true

- name: Create STAC Catalog
uses: docker://ghcr.io/linz/argo-tasks:v4
with:
args: stac-catalog --output stac/catalog.json --template template/catalog.json /github/workspace/stac/

# Sync STAC files only on push to 'master'
- name: Sync STAC
if: ${{ !contains(github.event.head_commit.message, '[skip-sync]')}}
Expand Down

0 comments on commit 1e21ade

Please sign in to comment.