For more information on how this repository was set up, check out this Confluence how-to page on how to bootstrap AWS accounts (access limited to Sage Bionetworks staff).
There are important details on how the infrastructure is set up and how this repository is organized under the following sections, such as AWS Accounts and Repository Organization.
You must first install pipenv and the AWS CLI.
For testing, you can use the Administrator IAM role on the nextflow-dev AWS account. If you don't have access to this role, ask a Tower admin to request that you be granted permission. While using the nextflow-dev admin role, you should only deploy the stacks within the common and *-dev folders (see Configuration). The instructions below assume that you have already configured your IAM role with the AWS CLI.
# Create directory for remote sceptre templates
mkdir -p templates/remote/
# Install dependencies in an isolated virtual environment
pipenv install --dev
# Install pre-commit hooks into Git
pipenv run pre-commit installIf your text editor (e.g. Visual Studio Code) or shell (e.g. using direnv) can automatically activate the pipenv virtual environment, you can omit the pipenv shell command.
# Define the AWS CLI profile you want to use for testing
export AWS_PROFILE="<profile-name>"
# Test the deployment of a specific 'develop' stack
pipenv run sceptre --var-file ./src/sceptre/variables/dev.yaml launch --yes infra-dev/some-stack.yaml
# Delete the test deployment of the specific 'develop' stack
pipenv run sceptre --var-file ./src/sceptre/variables/dev.yaml delete --yes infra-dev/some-stack.yamlTwo AWS accounts are managed by this repository, both of which were bootstrapped using org-formation. They are defined in this organization.yaml file. The two accounts are:
workflows-nextflow-prod(ornextflow-prod) for production useworkflows-nextflow-dev(ornextflow-dev) for testing purposesorg-sagebase-strides-ampad-workflows(orstrides-ampad) for production use within the AMP-AD project (AWS bills directly to the NIH) and should be treated likenextflow-prod
In general, we use the nextflow-dev account for testing templates before deploying them into the nextflow-prod account. New template configurations don't necessarily need to go through nextflow-dev first. For example, new instances of the tower-project.j2 template don't need to be deployed in nextflow-dev first.
Human access to these AWS accounts (programmatically or via the Management Console) is provided using IAM roles defined in organizations-infra (search for "WorkflowsNextflow"). These roles are individually bound to user groups in JumpCloud, which are managed by Sage IT.
- TowerViewer: This role is available in both
nextflow-devandnextflow-prodand grants access to the relevant subset of permissions inViewOnlyAccess. - Developer-IAM-Contrained: This role is only available in
nextflow-devfor testing and development purposes and is attached to thePowerUserAccessmanaged policy and some IAM access, i.e. you can only create/edit/delete IAM resources that have thedev-prefix in their names, so that important IAM resources aren't disrupted. - Administrator: This role exists in both the
nextflow-devandnextflow-prodaccounts (with separate user groups in JumpCloud) and grants full access, i.e.AdministratorAccess.
This repository follows the standard Sceptre directory structure.
Bespoke CloudFormation templates are stored in the templates/ directory, whereas general-purpose templates are obtained from the aws-infra repository during deployment. These external templates are downloaded into the templates/remote/ subdirectory using pre-launch Sceptre hooks (example).
Instances of each template are configured to be deployed in the config/ directory, which is organized into subfolders known as stack groups. In this case, we have defined several stack groups: common, infra-*, and projects-*. The CI/CD workflow deploys these stack groups to the AWS accounts as follows (in the listed order):
commonand*-devto thenextflow-devaccountcommonand*-prodto thenextflow-prodaccountcommonand*-ampadto thestrides-ampadaccount
There are two types of stack group configurations used in this repository. First, we have a shared config/config.yaml configuration, which contains values that are applicable to all stack groups. Second, we have configurations that are specific to individual stack groups, which define account-specific values (like IAM role ARNs).
GitHub Actions are used for continuous integration and deployment (CI/CD). Currently, this repository is configured with a single workflow, rw-deploy, which runs some lint checks and conditionally deploys the stacks on pushes to the main branch.
The lint checks are defined as pre-commit hooks and are partially configured by the .yamllint file. See above for how to locally set up pre-commit hooks in Git to ensure that these checks are run before every commit.
The CI/CD workflow uses a matrix strategy in conjunction with repository environments. See below for more information on secrets managements.
N.B. If the CI/CD workflow fails (either due to new commits or earlier commits), some changes might not get deployed. You can check the state of deployments from the main branch here.
This repository uses the Pipenv Python package to manage dependencies. The main dependencies and their required versions (if applicable) are listed in the Pipfile whereas the Pipfile.lock lists all recursive dependencies, their versions, and their checksums at the time of generating the lockfile. This Pipenv environment is used for CI/CD, but it can also be used for local development and testing. See above for how to set up a local development environment using Pipenv.
Additional dependencies exist for the pre-commit hooks that we've added to this repository. The virtual environments for these hooks are automatically configured when you run pre-commit.
bin/mirror-igenomes.sh: This Bash script is manually run to synchronize a subset of human and mouse reference files from the nf-core iGenomes bucket to a Sage-owned bucket inus-east-1. There are multiple reasons: (1) Sage AWS accounts cannot make requests to regions outside of the US; (2) we don't want nf-core to incur egress charges on our behalf; (3) creating a local mirror inus-east-1should reduce latency; and (4) we own a copy of the reference files in case anything happens to the nf-core bucket (e.g. AWS funding being cut). At the moment, this script is intended to be run manually because reference files shouldn't change all that often and thus automating the process wasn't deemed worth the effort.
The CI/CD workflow and Sceptre configurations make use of the following secrets.
After switching to a matrix strategy in the CI/CD workflow, all secrets are being stored in repository environments (see below). Organization secrets are no longer being used.
Our GitHub secrets are stored in account-specific environments. These secrets contain AWS and Nextflow Tower credentials.
The Github OIDC for GH actions were bootstrapped in the organizations-infra repository. For nextflow-dev and nextflow-prod, look for GithubOidcWorkflowsDevNextflowInfra in this file. For strides-ampad, look at this file.
The Nextflow Tower credentials (i.e. access tokens) were created manually using the Google service accounts that we provisioned for creating the Google OAuth clients, [email protected] and [email protected], respectively. The login info for these two Google accounts are stored in LastPass under the Shared-IBC-DPE-Workflows folder. Note that the *-prod and *-ampad stacks are configured to use the token associated with [email protected] whereas the *-dev stacks are configured to use the token associated with [email protected].
TOWER_TOKEN: The Nextflow Tower access token that will be used to provision the Tower teams, workspaces, credentials, and compute environments.
The following secrets were manually created in AWS Secrets Manager. They are used in some Sceptre configurations and are retrieved using a Sceptre resolver. We also create secrets during deployment and store them in Secrets Manager (example), but these aren't listed here. These are encrypted using the workflows-infra KMS key.
nextflow/license: The paid license key for Nextflow Towernextflow/google_oauth_app: The Google OAuth client credentialsnextflow/github_service_acct: The GitHub service account credentialsnextflow/synapse_oauth_client: The Synapse OAuth client ID and secret
The following secrets were created in all AWS accounts (including strides-ampad). Note that personal access tokens were used as password where possible (see notes in LastPass). Each of these secrets were created with two key-value pairs (username and password). Another service account (nextflow/ecr_service_acct) is created using CloudFormation (see wave-ecr-user stack).
nextflow/dockerhub_service_acct: The Docker Hub service account credentials for the Wave servicenextflow/ghcr_service_acct: The GHCR service account credentials for the Wave servicenextflow/quayio_service_acct: The Quay.io service account credentials for the Wave service
After a new deployment has successfully completed, it is important to ensure things are working as expected by doing the following:
- Launch a simple workflow such as
nextflow-io/hellofrom the UI using bothspotandon-demandcompute environments. - Run the
demo.pyscript from thepy-orcarepository. Make sure that your connection URI environment variable points to the correct URL and workspace. This will check that the API is working as expected and that individual workspaces are able to access their associated S3 buckets.
- The CIDR ranges of IP addresses specifies in the VPC configurations were added to the Sage VPN table.