Integrate OpenMetadata with the platform

[MattTriano]

Following an exploration of open source metadata platform options, in #35 , I'm going to go with OpenMetadata.

I'll want to modify the OpenMetadata ingestion and server frameworks to use my existing Airflow setup rather than have a second Airflow setup just for running OpenMetadata, and they kindly provide instructions.

• Set up Airflow lineage backend
• Add Connectors for any sources I want to ingest metadata from (and add their DAGs to the DagBag)
• Add Airflow API endpoints so that OpenMetadata's UI can communicate ingestion requests to Airflow services.
• Extract the MOUNTAIN OF ENV VARS in their docker-compose into dot-env files.

[MattTriano]

Handle Env Vars:

openmetadata-server

• AUTHORIZER_CLASS_NAME: ${AUTHORIZER_CLASS_NAME:-org.openmetadata.service.security.DefaultAuthorizer}
• AUTHORIZER_REQUEST_FILTER: ${AUTHORIZER_REQUEST_FILTER:-org.openmetadata.service.security.JwtFilter}
• AUTHORIZER_ADMIN_PRINCIPALS: ${AUTHORIZER_ADMIN_PRINCIPALS:-[admin]}
• AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN: ${AUTHORIZER_ALLOWED_REGISTRATION_DOMAIN:-["all"]}
• AUTHORIZER_INGESTION_PRINCIPALS: ${AUTHORIZER_INGESTION_PRINCIPALS:-[ingestion-bot]}
• AUTHORIZER_PRINCIPAL_DOMAIN: ${AUTHORIZER_PRINCIPAL_DOMAIN:-"openmetadata.org"}
• AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN: ${AUTHORIZER_ENFORCE_PRINCIPAL_DOMAIN:-false}
• AUTHORIZER_ENABLE_SECURE_SOCKET: ${AUTHORIZER_ENABLE_SECURE_SOCKET:-false}
• AUTHENTICATION_PROVIDER: ${AUTHENTICATION_PROVIDER:-basic}
• CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME: {CUSTOM_OIDC_AUTHENTICATION_PROVIDER_NAME:-""}
• AUTHENTICATION_PUBLIC_KEYS: ${AUTHENTICATION_PUBLIC_KEYS:-[http://localhost:8585/api/v1/config/jwks]}
• AUTHENTICATION_AUTHORITY: ${AUTHENTICATION_AUTHORITY:-https://accounts.google.com}
• AUTHENTICATION_CLIENT_ID: ${AUTHENTICATION_CLIENT_ID:-""}
• AUTHENTICATION_CALLBACK_URL: ${AUTHENTICATION_CALLBACK_URL:-""}
• AUTHENTICATION_JWT_PRINCIPAL_CLAIMS: ${AUTHENTICATION_JWT_PRINCIPAL_CLAIMS:-[email,preferred_username,sub]}
• AUTHENTICATION_ENABLE_SELF_SIGNUP: ${AUTHENTICATION_ENABLE_SELF_SIGNUP:-true}
• # JWT Configuration
• RSA_PUBLIC_KEY_FILE_PATH: ${RSA_PUBLIC_KEY_FILE_PATH:-"./conf/public_key.der"}
• RSA_PRIVATE_KEY_FILE_PATH: ${RSA_PRIVATE_KEY_FILE_PATH:-"./conf/private_key.der"}
• JWT_ISSUER: ${JWT_ISSUER:-"open-metadata.org"}
• JWT_KEY_ID: ${JWT_KEY_ID:-"Gb389a-9f76-gdjs-a92j-0242bk94356"}
• # OpenMetadata Server Airflow Configuration
• AIRFLOW_HOST: ${AIRFLOW_HOST:-http://ingestion:8080}
• SERVER_HOST_API_URL: ${SERVER_HOST_API_URL:-http://openmetadata-server:8585/api}
• #Database configuration for postgresql
• DB_DRIVER_CLASS: ${DB_DRIVER_CLASS:-org.postgresql.Driver}
• DB_SCHEME: ${DB_SCHEME:-postgresql}
• DB_USE_SSL: ${DB_USE_SSL:-false}
• DB_USER: ${DB_USER:-openmetadata_user}
• DB_USER_PASSWORD: ${DB_USER_PASSWORD:-openmetadata_password}
• DB_HOST: ${DB_HOST:-postgresql}
• DB_PORT: ${DB_PORT:-5432}
• OM_DATABASE: ${OM_DATABASE:-openmetadata_db}
• # Airflow SSL Configurations
• AIRFLOW_VERIFY_SSL: ${AIRFLOW_VERIFY_SSL:-"no-ssl"}
• AIRFLOW_SSL_CERT_PATH: ${AIRFLOW_SSL_CERT_PATH:-""}
• # ElasticSearch Configurations
• ELASTICSEARCH_HOST: ${ELASTICSEARCH_HOST:- elasticsearch}
• ELASTICSEARCH_PORT: ${ELASTICSEARCH_PORT:-9200}
• ELASTICSEARCH_SCHEME: ${ELASTICSEARCH_SCHEME:-http}
• ELASTICSEARCH_USER: ${ELASTICSEARCH_USER:-""}
• ELASTICSEARCH_PASSWORD: ${ELASTICSEARCH_PASSWORD:-""}
• # Heap OPTS Configurations
• OPENMETADATA_HEAP_OPTS: ${OPENMETADATA_HEAP_OPTS:--Xmx1G -Xms1G}

Ingestion (will need special handling for altered airflow setup)

• AIRFLOW__API__AUTH_BACKENDS: "airflow.api.auth.backend.basic_auth,airflow.api.auth.backend.session"
• AIRFLOW__CORE__EXECUTOR: LocalExecutor
• AIRFLOW__OPENMETADATA_AIRFLOW_APIS__DAG_GENERATED_CONFIGS: "/opt/airflow/dag_generated_configs"
• DB_HOST: ${AIRFLOW_DB_HOST:-postgresql}
• DB_PORT: ${AIRFLOW_DB_PORT:-5432}
• AIRFLOW_DB: ${AIRFLOW_DB:-airflow_db}
• DB_USER: ${AIRFLOW_DB_USER:-airflow_user}
• DB_SCHEME: ${AIRFLOW_DB_SCHEME:-postgresql+psycopg2}
• DB_PASSWORD: ${AIRFLOW_DB_PASSWORD:-airflow_pass}

Airflow Lineage

• AIRFLOW__LINEAGE__BACKEND="airflow_provider_openmetadata.lineage.backend.OpenMetadataLineageBackend"
• AIRFLOW__LINEAGE__AIRFLOW_SERVICE_NAME="local_airflow"
• AIRFLOW__LINEAGE__OPENMETADATA_API_ENDPOINT="http://localhost:8585/api"
• AIRFLOW__LINEAGE__AUTH_PROVIDER_TYPE="openmetadata"
• AIRFLOW__LINEAGE__JWT_TOKEN="" (I think this is just related to SSO and the interface might generate the token)

[MattTriano]

Resources referenced:

• Elasticsearch password add
• Configuring Elasticsearch with Docker

Auth-related

• Basic auth config (has info on generating public and private .der keys and on the JWT_KEY_ID (any old UUID))
• JSON Web Tokens (JWT) purpose
• JWT Config
• Generating a JWT token in the UI

Docker Compose

• dot-env file parsing behavior