03-orchestration

3. Orchestration and ML Pipelines

This section of the repo contains Python code to accompany the videos that show how to use Prefect for MLOps. We will create workflows that you can orchestrate and observe.

3.1 Introdution to Workflow Orchestration

3.2 Introduction to Prefect

3.3 Prefect Workflow

3.4 Deploying Your Workflow

3.5 Working with Deployments

3.6 Prefect Cloud (optional)

3.7 Homework

More information here.

Quick setup

Install packages

In a conda environment with Python 3.10.12 or similar, install all package dependencies with

pip install -r requirements.txt

Start the Prefect server locally

Create another window and activate your conda environment. Start the Prefect API server locally with

prefect server start

Alternative to self-hosted server use Prefect Cloud for added capabilties

Signup and use for free at https://app.prefect.cloud

Authenticate through the terminal with

prefect cloud login

Use your Prefect profile to switch between a self-hosted server and Cloud.

Notes

Did you take notes? Add them here:

• Prefect local deployment using Google Cloud Storage as data storage
• Notes/Blog posts by Sagar Thacker
  • Inrtoduction to Workflow Orchestratio & Prefect
  • Prefect Blocks
  • Prefect Deployments
  • Prefect Cloud Deployment
• Week 3: Prefect by Hongfan (Amber)
• Send a PR, add your notes above this line

Notes 2022 Edition

To read the notes from the previous edition, see here