This repository contains various configuration files, tools and utilities for operating Metaflow in production. See Metaflow documentation for more information about Metaflow architecture. Top level folders are structured as follows:
If you're not already using Terraform, this is the easiest way to get started on AWS. You can find the template under aws/cloudformation in this repository. This stack uses AWS Batch for compute and AWS Step Functions for orchestration.
Another deployment option is using Terraform. An end-to-end example can be found under aws/terraform in this repository. The example leverages the official Terraform module outerbounds/terraform-aws-metaflow as a building block. This stack uses AWS Batch for compute and AWS Step Functions for orchestration.
This is the quickest way to spin up a fully functional Metaflow stack on Azure. See details under azure/terraform in this repository. This stack uses Kubernetes (AKS) for compute and Argo Workflows for orchestration.
We provide Helm charts to deploy Metaflow Metadata service and UI in a K8S cluster. This way you can use Metaflow without any AWS-specific dependencies on AWS except for having a S3-compatible object storage engine available. You can find them under k8s/helm/metaflow in this repository.
Some Metaflow flows that could be used to test drive a Metaflow stack. Some of these flows are used to drive end-to-end CI coverage internally at Outerbounds. They live under common/sample_flows
Scripts that make life easier either deploying or using your new Metaflow stacks.
Talk to us on Slack.