Dagster Cloud CI/CD GitHub Action

GitHub Action to update Dagster Cloud code locations, building and pushing Docker images when pipeline code is updated.

Quickstart

Want to get started right away, or look at a functional example for reference? We provide a quickstart template repo which you can use to get CI for your Cloud instance up and running quickly.

Usage

This action requires that access to the target Docker registry is set up, and that the Git repository is cloned using the actions/checkout action.

The action utilizes a locations.yaml file which describes each of the Dagster Cloud repo locations to be built and updated. If this locations.yaml file is not located at the repo root, it must be specified with the location-file input.

Example Job

This example uses the docker/login-action action to set up Docker registry access. ECR users may want to use the aws-actions/amazon-ecr-login action instead. To speed up Docker builds, you may also use the satackey/action-docker-layer-caching action.

on:
  push:
    branches:
      - main

jobs:
  dagster-cloud:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout repo
        uses: actions/checkout@v1

      - name: Login to DockerHub
        uses: docker/login-action@v1
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build images & update Dagster Cloud
        uses: dagster-io/[email protected]
        with:
          dagit-url: https://hooli.dagster.cloud/prod
          api-token: ${{ secrets.DAGSTER_AGENT_TOKEN }}

Example locations.yaml

This locations file indicates that two locations, foo and bar, should be built. These locations have Dockerfiles located at /foo_src/Dockerfile and /bar_src/Dockerfile, and are pushed to the dagster-io/foo and dagster-io/bar registries, respectively.

locations:
  # Location name
  foo:

    # Path to build directory, which must contain a Dockerfile or
    # requirements.txt file, relative to the locations.yaml folder
    build: ./foo_src

    # The base Docker image to use, if providing only a requirements.txt
    # file and no Dockerfile
    base_image: python:3.8-slim

    # Docker registry to push the built Docker image to
    registry: dagster-io/foo

    # Python file containing the job repo
    # Can alternatively supply python_module, as below
    python_file: repo.py

  bar:
    build: ./bar_src
    registry: dagster-io/bar
    python_module: bar

More Examples

More examples are provided in the example folder.

Customization

Inputs

Name	Description
dagit-url	(Required) URL to your Dagit Cloud instance, including the deployment path.
api-token	(Required) Dagster Cloud Agent API token.
location-file	Path to the locations.yaml file defining the code locations to update. Defaults to /locations.yaml in the repo root.
image-tag	Tag for the built Docker images, defaults to the first 6 chars of git hash.
parallel	Whether to build and push Docker images in parallel. Defaults to true.

Developing the CI/CD Action

The CI/CD action is run from the packaged files in the dist/* folder. When making a change, be sure to repackage the files:

npm run prepare