If you're looking for Airflow videos from the 2022 edition, check the 2022 cohort folder.
- What is a Data Lake
- ELT vs. ETL
- Alternatives to components (S3/HDFS, Redshift, Snowflake etc.)
- Video
- Slides
- What is orchestration?
- Workflow orchestrators vs. other types of orchestrators
- Core features of a workflow orchestration tool
- Different types of workflow orchestration tools that currently exist
🎥 Video
- What is Prefect?
- Installing Prefect
- Prefect flow
- Creating an ETL
- Prefect task
- Blocks and collections
- Orion UI
🎥 Video
- Flow 1: Putting data to Google Cloud Storage
🎥 Video
- Flow 2: From GCS to BigQuery
🎥 Video
- Parametrizing the script from your flow
- Parameter validation with Pydantic
- Creating a deployment locally
- Setting up Prefect Agent
- Running the flow
- Notifications
🎥 Video
- Scheduling a deployment
- Flow code storage
- Running tasks in Docker
🎥 Video
- Using Prefect Cloud instead of local Prefect
- Workspaces
- Running flows on GCP
🎥 Video
Code from videos (with a few minor enhancements)
To be linked here by Jan. 30
Did you take notes? You can share them here.
- Blog by Marcos Torregrosa (Prefect)
- Add your notes here (above this line)
Most of these notes are about Airflow, but you might find them useful.