Merge pull request #5 from michaelconan/feat/startup

feat: add config file, script adjustments, compose variables

Author: michaelconan

.devcontainer/docker-compose.yml

@@ -2,9 +2,15 @@ version: '3.8'
 
 services:
   app:
+    env:
+      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://postgres:postgres@db/postgres
+      AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
+      AIRFLOW__CORE__FERNET_KEY: ""
+      AIRFLOW__CORE__EXECUTOR: LocalExecutor
+      AIRFLOW__CORE__LOAD_EXAMPLES: "true"
+      AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION: "true"
     build:
-      # Use Airflow Dockerfile
-      context: ..
+      context: .
       dockerfile: Dockerfile
 
     volumes:

.gitattributes

@@ -1,2 +1,3 @@
 * text=auto
-*.py eol=lf
+*.py eol=lf
+script/* eol=lf

README.md

@@ -23,21 +23,40 @@ I have found a few resources covering deploying Airflow on Azure, none of which
 - DAGs should contain a number of related steps (e.g., extract-transform-load)
 - DAGs can be linked to one another via datasets to enable triggering and dependency graph
 
-## Airflow Setup
+## Python Setup (Attempt #2)
+
+As an alternative to Docker I tried a standard Python configuration for easier deployment to App Service.
+
+### Airflow Setup
+
+### Azure Setup
+
+1. Create Web App + PostgreSQL with Python
+2. Turn on Application Insights, Logging
+
+### Automated Deployment
+
+1. Referenced [this workflow](https://learn.microsoft.com/en-us/azure/app-service/deploy-github-actions?tabs=applevel%2Cpython%2Cpythonn) to deploy Python app to App Service
+
+## Docker Setup (Attempt #1)
+
+This was working fine locally but deployment to App Service did not go as well. Goal to use a custom extended airflow Docker container
+
+### Airflow Setup
 
 1. Started with official [docker compose](https://airflow.apache.org/docs/apache-airflow/stable/howto/docker-compose/index.html)
 2. Decided to [extend the image](https://airflow.apache.org/docs/docker-stack/build.html#extending-the-image) starting with the slim version due to limited packaged requirements
 3. Switched to `LocalExecutor` 
 4. Stripped down the Docker Compose to a single container to run webserver + scheduler and use for dev container
 
-## Azure Setup
+### Azure Setup
 
 From the resources above I determined I can run the webserver and scheduler together in a single container with `LocalExecutor`. `CeleryExecutor` shouldn't be necessary unless scale increases and multiple workers are required.
 
 1. Create PostgreSQL flexible server for database
 2. Create App Service app with container (NOTE: I followed [this tutorial](https://learn.microsoft.com/en-us/azure/app-service/configure-custom-container?tabs=debian&pivots=container-linux#enable-ssh) to configure SSH access in app service)
 
-# Automated Deployment
+### Automated Docker Deployment
 
 1. Referenced [this workflow](https://docs.github.com/en/actions/use-cases-and-examples/publishing-packages/publishing-docker-images#publishing-images-to-github-packages) to build and publish to GitHub Container Registry
 2. Referenced [this workflow](https://learn.microsoft.com/en-us/azure/app-service/deploy-best-practices#use-github-actions) to deploy updated image to Azure Web App

script/setup

@@ -13,5 +13,5 @@ script/bootstrap
 echo "==> Setting up DB…"
 
 # Reset Airflow database
-airflow db downgrade base
+airflow db reset --yes
 airflow db migrate

webserver_config.py

@@ -0,0 +1,7 @@
+"""
+webserver_config.py
+
+Configuration of Airflow webserver and definition of Flask app object,
+which may be beneficial for Azure app service custom startup.
+"""
+from flask import current_app as app