Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: ensure only the latest stac documents are synced into s3 #300

Merged
merged 2 commits into from
Jun 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
### Motivation

<!-- TODO: Say why you made your changes. -->

### Modifications

<!-- TODO: Say what changes you made. -->

<!-- TODO: Attach screenshots if you changed the UI. -->

### Verification

<!-- TODO: Say how you tested your changes. -->
39 changes: 34 additions & 5 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -1,74 +1,93 @@
name: Publish
on: [push]
on:
push:
workflow_dispatch:

jobs:
main:
name: Validate STAC
runs-on: ubuntu-latest
steps:
- uses: linz/action-typescript@dee99184c4305aea6c380a52db9b2d7abaaa3e78 # v3
- uses: linz/action-typescript@9bf69b0f313b3525d3ba3116f26b1aff7eb7a6c0 # v3
with:
node-version: 20.x
# FIXME: catalog.json is not pushed to the repository (temporary solution)

- name: Create STAC Catalog
uses: docker://ghcr.io/linz/argo-tasks:v4
with:
args: stac-catalog --output stac/catalog.json --template template/catalog.json /github/workspace/stac/

- name: Validate STAC Catalog
uses: docker://ghcr.io/linz/argo-tasks:v4
with:
args: stac-validate /github/workspace/stac/catalog.json

- name: Validate STAC Collections
run: |
# Enable double star operator
shopt -s globstar
docker run -v "$PWD:$PWD" ghcr.io/linz/argo-tasks:v4 stac-validate "$PWD"/stac/**/collection.json

- name: Download actionlint
run: docker build --tag actionlint - < .github/workflows/actionlint.dockerfile

- name: Run actionlint to check workflow files
run: docker run --volume="${PWD}:/repo" --workdir=/repo actionlint -color

publish-odr:
name: Publish ODR
runs-on: ubuntu-latest
needs: main
if: ${{ github.ref == 'refs/heads/master' }}

environment:
name: prod

permissions:
id-token: write
contents: read

steps:
- uses: linz/action-typescript@dee99184c4305aea6c380a52db9b2d7abaaa3e78 # v3
- uses: linz/action-typescript@9bf69b0f313b3525d3ba3116f26b1aff7eb7a6c0 # v3
with:
node-version: 20.x

- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
with:
fetch-depth: 0

- name: Use Node.js 18.x
uses: actions/setup-node@8f152de45cc393bb48ce5d89d36b731f54556e65 # v4.0.0
with:
node-version: '18.x'

- name: Setup kubectl
uses: azure/setup-kubectl@901a10e89ea615cf61f57ac05cecdf23e7de06d8 # v3
uses: azure/setup-kubectl@3e0aec4d80787158d308d7b364cb1b702e7feb7f # v4.0.0
with:
version: 'latest'

- name: AWS Configure
uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4
with:
aws-region: ap-southeast-2
mask-aws-account-id: true
role-to-assume: ${{ secrets.AWS_EKS_CI_ROLE }}

- name: Login to EKS
run: |
aws eks update-kubeconfig --name Workflows --region ap-southeast-2

- name: Check EKS connection
run: |
kubectl get nodes

- name: Install Argo
run: |
curl -sLO https://github.com/argoproj/argo-workflows/releases/download/v3.4.11/argo-linux-amd64.gz
gunzip argo-linux-amd64.gz
chmod +x argo-linux-amd64
./argo-linux-amd64 version

- name: Submit Added/Changed Parameter Files
id: modified-files
if: ${{ !contains(github.event.head_commit.message, '[skip-argo]')}}
Expand All @@ -79,36 +98,46 @@ jobs:
for file in "${modified_parameter_files[@]}"; do
./argo-linux-amd64 submit --wait --from wftmpl/copy -n argo -f "$file" -p aws_role_config_path="s3://linz-bucket-config/config-write.open-data-registry.json" -p exclude="collection.json$" --generate-name "publish-odr-file-copy-"
done

sync-stac:
name: Sync STAC files
runs-on: ubuntu-latest
concurrency: publish-${{ github.ref }}
needs: publish-odr
if: ${{ github.ref == 'refs/heads/master' }}

environment:
name: prod

permissions:
id-token: write
contents: read

steps:
- uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
with:
ref: master # Ensure only the latest STAC documents are synced into S3

- name: AWS Configure
uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4
with:
aws-region: ap-southeast-2
mask-aws-account-id: true
role-to-assume: ${{ secrets.AWS_ODR_CI_ROLE }}

- name: AWS Configure ODR
uses: aws-actions/configure-aws-credentials@010d0da01d0b5a38af31e9c3470dbfdabdecca3a # v4
with:
aws-region: ap-southeast-2
mask-aws-account-id: true
role-to-assume: ${{ secrets.AWS_ODR_DATA_MANAGER_ROLE }}
role-chaining: true

- name: Create STAC Catalog
uses: docker://ghcr.io/linz/argo-tasks:v4
with:
args: stac-catalog --output stac/catalog.json --template template/catalog.json /github/workspace/stac/

# Sync STAC files only on push to 'master'
- name: Sync STAC
if: ${{ !contains(github.event.head_commit.message, '[skip-sync]')}}
Expand Down