diff --git a/.gitignore b/.gitignore index 58a3109..67f095d 100644 --- a/.gitignore +++ b/.gitignore @@ -115,6 +115,7 @@ celerybeat.pid # Environments .env .venv +.venvclear/ env/ venv/ ENV/ diff --git a/CHANGELOG.md b/CHANGELOG.md index e674942..ac28ca0 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,10 @@ # Changelog +## [v.0.0.7] +- Added dlt-meta cli documentation and readme with browser support: [PR](https://github.com/databrickslabs/dlt-meta/pull/45) + +## [v.0.0.6] +- migrate to create streaming table api from create streaming live table: [PR](https://github.com/databrickslabs/dlt-meta/pull/39) ## [v.0.0.5] - Enabled Unity Catalog support: [PR](https://github.com/databrickslabs/dlt-meta/pull/28) diff --git a/README.md b/README.md index f136403..8fb0273 100755 --- a/README.md +++ b/README.md @@ -2,18 +2,21 @@ + [Documentation](https://databrickslabs.github.io/dlt-meta/) | [Release Notes](CHANGELOG.md) | -[Examples](https://github.com/databrickslabs/dlt-meta/tree/main/examples) +[Examples](https://github.com/databrickslabs/dlt-meta/tree/main/examples) + --- +

Documentation Status - Latest Python Release + Latest Python Release codecov - - lgtm-alerts - - - lgtm-code-quality - downloads @@ -39,142 +35,141 @@

-[![lines of code](https://tokei.rs/b1/github/databrickslabs/dlt-meta)]([https://codecov.io/github/databrickslabs/dlt-meta](https://github.com/databrickslabs/dlt-meta)) +[![lines of code](https://tokei.rs/b1/github/databrickslabs/dlt-meta)](<[https://codecov.io/github/databrickslabs/dlt-meta](https://github.com/databrickslabs/dlt-meta)>) --- # Project Overview -```DLT-META``` is a metadata-driven framework based on Databricks [Delta Live Tables](https://www.databricks.com/product/delta-live-tables) (aka DLT) which lets you automate your bronze and silver data pipelines. -With this framework you need to record the source and target metadata in an onboarding json file which acts as the data flow specification aka Dataflowspec. A single generic ```DLT``` pipeline takes the ```Dataflowspec``` and runs your workloads. +`DLT-META` is a metadata-driven framework based on Databricks [Delta Live Tables](https://www.databricks.com/product/delta-live-tables) (aka DLT) which lets you automate your bronze and silver data pipelines. + +With this framework you need to record the source and target metadata in an onboarding json file which acts as the data flow specification aka Dataflowspec. A single generic `DLT` pipeline takes the `Dataflowspec` and runs your workloads. ### Components: -#### Metadata Interface +#### Metadata Interface + - Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json) - Capture [Data Quality Rules](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe/customers/bronze_data_quality_expectations.json) -- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json) +- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json) #### Generic DLT pipeline + - Apply appropriate readers based on input metadata -- Apply data quality rules with DLT expectations +- Apply data quality rules with DLT expectations - Apply CDC apply changes if specified in metadata - Builds DLT graph based on input/output metadata - Launch DLT pipeline ## High-Level Process Flow: + ![DLT-META High-Level Process Flow](./docs/static/images/solutions_overview.png) ## Steps + ![DLT-META Stages](./docs/static/images/dlt-meta_stages.png) ## Getting Started + Refer to the [Getting Started](https://databrickslabs.github.io/dlt-meta/getting_started) + ### Databricks Labs DLT-META CLI lets you run onboard and deploy in interactive python terminal + #### pre-requisites: -- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html) + - Python 3.8.0 + -#### Steps: -- ``` git clone dlt-meta ``` -- ``` cd dlt-meta ``` -- ``` python -m venv .venv ``` -- ```source .venv/bin/activate ``` -- ``` pip install databricks-sdk ``` -- ```databricks labs dlt-meta onboard``` -- - Above command will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder. - -``` Provide onboarding file path (default: demo/conf/onboarding.template): - Provide onboarding files local directory (default: demo/): - Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): - Provide databricks runtime version (default: 14.2.x-scala2.12): - Run onboarding with unity catalog enabled? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide unity catalog name: ravi_dlt_meta_uc - Provide dlt meta schema name (default: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead): - Provide dlt meta bronze layer schema name (default: dltmeta_bronze_cf5956873137432294892fbb2dc34fdb): - Provide dlt meta silver layer schema name (default: dltmeta_silver_5afa2184543342f98f87b30d92b8c76f): - Provide dlt meta layer - [0] bronze - [1] bronze_silver - [2] silver - Enter a number between 0 and 2: 1 - Provide bronze dataflow spec table name (default: bronze_dataflowspec): - Provide silver dataflow spec table name (default: silver_dataflowspec): - Overwrite dataflow spec? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide dataflow spec version (default: v1): - Provide environment name (default: prod): prod - Provide import author name (default: ravi.gawai): - Provide cloud provider name - [0] aws - [1] azure - [2] gcp - Enter a number between 0 and 2: 0 - Do you want to update ws paths, catalog, schema details to your onboarding file? - [0] False - [1] True + +- Databricks CLI v0.213 or later. See [instructions](https://docs.databricks.com/en/dev-tools/cli/tutorial.html) + +- Install Databricks CLI on macOS: +- ![macos_install_databricks](docs/static/images/macos_1_databrickslabsmac_installdatabricks.gif) + +- Install Databricks CLI on Windows: +- ![windows_install_databricks.png](docs/static/images/windows_install_databricks.png) + +Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace: + +```commandline +databricks auth login --host WORKSPACE_HOST +``` + + To enable debug logs, simply add `--debug` flag to any command. + +### Installing dlt-meta: + +- Install dlt-meta via Databricks CLI: + +```commandline + databricks labs install dlt-meta +``` + +### Onboard using dlt-meta CLI: + +If you want to run existing demo files please follow these steps before running onboard command: + +```commandline + git clone https://github.com/databrickslabs/dlt-meta.git ``` + +```commandline + cd dlt-meta +``` + +```commandline + python -m venv .venv +``` + +```commandline + source .venv/bin/activate +``` + +```commandline + pip install databricks-sdk +``` + +```commandline + databricks labs dlt-meta onboard +``` + +![onboardingDLTMeta.gif](docs/static/images/onboardingDLTMeta.gif) + +Above commands will prompt you to provide onboarding details. If you have cloned dlt-meta git repo then accept defaults which will launch config from demo folder. +![onboardingDLTMeta_2.gif](docs/static/images/onboardingDLTMeta_2.gif) + + - Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs + +### depoly using dlt-meta CLI: + - Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command -- ```databricks labs dlt-meta deploy``` +- ```commandline + databricks labs dlt-meta deploy + ``` - - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps - - Bronze DLT -``` - Deploy DLT-META with unity catalog enabled? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide unity catalog name: ravi_dlt_meta_uc - Deploy DLT-META with serverless? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide dlt meta layer - [0] bronze - [1] silver - Enter a number between 0 and 1: 0 - Provide dlt meta onboard group: A1 - Provide dlt_meta dataflowspec schema name: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead - Provide bronze dataflowspec table name (default: bronze_dataflowspec): - Provide dlt meta pipeline name (default: dlt_meta_bronze_pipeline_2aee3eb837f3439899eef61b76b80d53): - Provide dlt target schema name: dltmeta_bronze_cf5956873137432294892fbb2dc34fdb -``` + +![deployingDLTMeta_bronze.gif](docs/static/images/deployingDLTMeta_bronze.gif) + - Silver DLT -- - ```databricks labs dlt-meta deploy``` +- - ```commandline + databricks labs dlt-meta deploy + ``` - - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps -``` - Deploy DLT-META with unity catalog enabled? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide unity catalog name: ravi_dlt_meta_uc - Deploy DLT-META with serverless? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide dlt meta layer - [0] bronze - [1] silver - Enter a number between 0 and 1: 1 - Provide dlt meta onboard group: A1 - Provide dlt_meta dataflowspec schema name: dlt_meta_dataflowspecs_203b9da04bdc49f78cdc6c379d1c9ead - Provide silver dataflowspec table name (default: silver_dataflowspec): - Provide dlt meta pipeline name (default: dlt_meta_silver_pipeline_2147545f9b6b4a8a834f62e873fa1364): - Provide dlt target schema name: dltmeta_silver_5afa2184543342f98f87b30d92b8c76f -``` + +![deployingDLTMeta_silver.gif](docs/static/images/deployingDLTMeta_silver.gif) + + ## More questions + Refer to the [FAQ](https://databrickslabs.github.io/dlt-meta/faq) and DLT-META [documentation](https://databrickslabs.github.io/dlt-meta/) # Project Support + Please note that all projects released under [`Databricks Labs`](https://www.databricks.com/learn/labs) - are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements -(SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket +are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements +(SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects. Any issues discovered through the use of this project should be filed as issues on the Github Repo. diff --git a/demo/README.md b/demo/README.md index bfaf91b..ba1d0a5 100644 --- a/demo/README.md +++ b/demo/README.md @@ -3,7 +3,8 @@ 2. [Databricks Techsummit Demo](#databricks-tech-summit-fy2024-demo): 100s of data sources ingestion in bronze and silver DLT pipelines automatically. -# DAIS 2023 DEMO +# DAIS 2023 DEMO +## [DAIS 2023 Session Recording](https://www.youtube.com/watch?v=WYv5haxLlfA) This Demo launches Bronze and Silver DLT pipleines with following activities: - Customer and Transactions feeds for initial load - Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes. @@ -23,7 +24,7 @@ This Demo launches Bronze and Silver DLT pipleines with following activities: export PYTHONPATH=<> ``` -6. Run the command ```python demo/launch_dais_demo.py --username=<> --source=cloudfiles --uc_catalog_name=<> --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-demo-automated_new``` +6. Run the command ```python demo/launch_dais_demo.py --source=cloudfiles --uc_catalog_name=<> --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-demo-automated_new``` - cloud_provider_name : aws or azure or gcp - db_version : Databricks Runtime Version - dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines @@ -61,7 +62,7 @@ This demo will launch auto generated tables(100s) inside single bronze and silve export PYTHONPATH=<> ``` -6. Run the command ```python demo/launch_techsummit_demo.py --username=ravi.gawai@databricks.com --source=cloudfiles --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/techsummit-dlt-meta-demo-automated ``` +6. Run the command ```python demo/launch_techsummit_demo.py --source=cloudfiles --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/techsummit-dlt-meta-demo-automated ``` - cloud_provider_name : aws or azure or gcp - db_version : Databricks Runtime Version - dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines diff --git a/demo/launch_dais_demo.py b/demo/launch_dais_demo.py index 42953f5..b73c460 100644 --- a/demo/launch_dais_demo.py +++ b/demo/launch_dais_demo.py @@ -1,4 +1,5 @@ import uuid +import webbrowser from databricks.sdk.service import jobs from src.install import WorkspaceInstaller from integration_tests.run_integration_tests import ( @@ -84,8 +85,10 @@ def launch_workflow(self, runner_conf: DLTMetaRunnerConf): runner_conf.job_id = created_job.job_id print(f"Job created successfully. job_id={created_job.job_id}, started run...") print(f"Waiting for job to complete. run_id={created_job.job_id}") - run_by_id = self.ws.jobs.run_now(job_id=created_job.job_id).result() - print(f"Job run finished. run_id={run_by_id}") + run_by_id = self.ws.jobs.run_now(job_id=created_job.job_id) + url = f"{self.ws.config.host}/jobs/{runner_conf.job_id}/runs/{run_by_id}?o={self.ws.get_workspace_id()}/" + webbrowser.open(url) + print(f"Job launched with url={url}") def create_daisdemo_workflow(self, runner_conf: DLTMetaRunnerConf): """ diff --git a/demo/launch_techsummit_demo.py b/demo/launch_techsummit_demo.py index be04576..3f59f39 100644 --- a/demo/launch_techsummit_demo.py +++ b/demo/launch_techsummit_demo.py @@ -23,6 +23,7 @@ """ import uuid +import webbrowser from databricks.sdk.service import jobs from databricks.sdk.service.catalog import VolumeType, SchemasAPI from databricks.sdk.service.workspace import ImportFormat @@ -163,8 +164,10 @@ def launch_workflow(self, runner_conf: DLTMetaRunnerConf): runner_conf.job_id = created_job.job_id print(f"Job created successfully. job_id={created_job.job_id}, started run...") print(f"Waiting for job to complete. run_id={created_job.job_id}") - run_by_id = self.ws.jobs.run_now(job_id=created_job.job_id).result() - print(f"Job run finished. run_id={run_by_id}") + run_by_id = self.ws.jobs.run_now(job_id=created_job.job_id) + url = f"{self.ws.config.host}/jobs/{runner_conf.job_id}/runs/{run_by_id}?o={self.ws.get_workspace_id()}/" + webbrowser.open(url) + print(f"Job launched with url={url}") def create_techsummit_demo_workflow(self, runner_conf: TechsummitRunnerConf): """ diff --git a/docs/config.toml b/docs/config.toml index 3c5a3dc..da3d64e 100644 --- a/docs/config.toml +++ b/docs/config.toml @@ -2,7 +2,6 @@ baseURL = 'https://databrickslabs.github.io/dlt-meta/' languageCode = 'en-us' title = 'DLT-META' theme= "hugo-theme-learn" - pluralizeListTitles = false canonifyURLs = true diff --git a/docs/content/getting_started/dltpipelineopt1.md b/docs/content/getting_started/dltpipelineopt1.md index 704dfcd..be97af4 100644 --- a/docs/content/getting_started/dltpipelineopt1.md +++ b/docs/content/getting_started/dltpipelineopt1.md @@ -10,13 +10,15 @@ draft: false - Python 3.8.0 + ##### Steps: ```shell - git clone dlt-meta + git clone https://github.com/databrickslabs/dlt-meta.git cd dlt-meta python -m venv .venv source .venv/bin/activate pip install databricks-sdk databricks labs dlt-meta onboard ``` + +![onboardingDLTMeta.gif](/images/onboardingDLTMeta.gif) - Once onboarding jobs is finished deploy `bronze` and `silver` DLT using below command #### Deploy Bronze DLT @@ -24,49 +26,13 @@ draft: false databricks labs dlt-meta deploy ``` - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps -```shell - Deploy DLT-META with unity catalog enabled? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide unity catalog name: uc_catalog_name - Deploy DLT-META with serverless? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide dlt meta layer - [0] bronze - [1] silver - Enter a number between 0 and 1: 0 - Provide dlt meta onboard group: A1 - Provide dlt_meta dataflowspec schema name: dlt_meta_dataflowspecs_203b9 - Provide bronze dataflowspec table name (default: bronze_dataflowspec): - Provide dlt meta pipeline name (default: dlt_meta_bronze_pipeline_2aee): - Provide dlt target schema name: dltmeta_bronze_cf595 -``` + +![deployingDLTMeta_bronze.gif](/images/deployingDLTMeta_bronze.gif) #### Deploy Silver DLT ```shell databricks labs dlt-meta deploy ``` - - Above command will prompt you to provide dlt details. Please provide respective details for schema which you provided in above steps -```shell - Deploy DLT-META with unity catalog enabled? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide unity catalog name: uc_catalog_name - Deploy DLT-META with serverless? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide dlt meta layer - [0] bronze - [1] silver - Enter a number between 0 and 1: 1 - Provide dlt meta onboard group: A1 - Provide dlt_meta dataflowspec schema name: dlt_meta_dataflowspecs_203b9 - Provide silver dataflowspec table name (default: silver_dataflowspec): - Provide dlt meta pipeline name (default: dlt_meta_silver_pipeline_21475): - Provide dlt target schema name: dltmeta_silver_5afa2 -``` \ No newline at end of file + +![deployingDLTMeta_silver.gif](/images/deployingDLTMeta_silver.gif) \ No newline at end of file diff --git a/docs/content/getting_started/runoboardingopt1.md b/docs/content/getting_started/runoboardingopt1.md index 1f9daf2..6840e6a 100644 --- a/docs/content/getting_started/runoboardingopt1.md +++ b/docs/content/getting_started/runoboardingopt1.md @@ -10,12 +10,14 @@ draft: false - [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html) - Python 3.8.0 + ##### Steps: -1. ``` git clone dlt-meta ``` +1. ``` git clone https://github.com/databrickslabs/dlt-meta.git ``` 2. ``` cd dlt-meta ``` 3. ``` python -m venv .venv ``` 4. ```source .venv/bin/activate ``` 5. ``` pip install databricks-sdk ``` +![onboardingDLTMeta.gif](/images/onboardingDLTMeta.gif) + ##### run dlt-meta cli command: ```shell databricks labs dlt-meta onboard @@ -24,41 +26,6 @@ draft: false - If you have cloned dlt-meta git repo then accepting defaults will launch config from [demo/conf](https://github.com/databrickslabs/dlt-meta/tree/main/demo/conf) folder. - You can create onboarding files e.g onboarding.json, data quality and silver transformations and put it in conf folder as show in [demo/conf](https://github.com/databrickslabs/dlt-meta/tree/main/demo/conf) -```shell - Provide onboarding file path (default: demo/conf/onboarding.template): - Provide onboarding files local directory (default: demo/): - Provide dbfs path (default: dbfs:/dlt-meta_cli_demo): - Provide databricks runtime version (default: 14.2.x-scala2.12): - Run onboarding with unity catalog enabled? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide unity catalog name: uc_catalog_name - Provide dlt meta schema name (default: dlt_meta_dataflowspecs_203b9): - Provide dlt meta bronze layer schema name (default: dltmeta_bronze_cf595): - Provide dlt meta silver layer schema name (default: dltmeta_silver_5afa2): - Provide dlt meta layer - [0] bronze - [1] bronze_silver - [2] silver - Enter a number between 0 and 2: 1 - Provide bronze dataflow spec table name (default: bronze_dataflowspec): - Provide silver dataflow spec table name (default: silver_dataflowspec): - Overwrite dataflow spec? - [0] False - [1] True - Enter a number between 0 and 1: 1 - Provide dataflow spec version (default: v1): - Provide environment name (default: prod): prod - Provide import author name (default: ravi.gawai): - Provide cloud provider name - [0] aws - [1] azure - [2] gcp - Enter a number between 0 and 2: 0 - Do you want to update ws paths, catalog, schema details to your onboarding file? - [0] False - [1] True -``` +![onboardingDLTMeta_2.gif](/images/onboardingDLTMeta_2.gif) - Goto your databricks workspace and located onboarding job under: Workflow->Jobs runs diff --git a/docs/content/getting_started/runoboardingopt3.md b/docs/content/getting_started/runoboardingopt3.md index c2c1f6a..c45e565 100644 --- a/docs/content/getting_started/runoboardingopt3.md +++ b/docs/content/getting_started/runoboardingopt3.md @@ -18,7 +18,6 @@ onboarding_params_map = { "silver_dataflowspec_table": "silver_dataflowspec_table", "silver_dataflowspec_path": "dbfs:/onboarding_tables_cdc/silver", "overwrite": "True", - "onboard_layer": "bronze_silver", "env": "dev", "version": "v1", "import_author": "Ravi" @@ -35,7 +34,6 @@ onboarding_params_map = { "bronze_dataflowspec_table": "bronze_dataflowspec_table", "silver_dataflowspec_table": "silver_dataflowspec_table", "overwrite": "True", - "onboard_layer": "bronze_silver", "env": "dev", "version": "v1", "import_author": "Ravi" diff --git a/docs/static/images/deployingDLTMeta_bronze.gif b/docs/static/images/deployingDLTMeta_bronze.gif new file mode 100644 index 0000000..de76dfa Binary files /dev/null and b/docs/static/images/deployingDLTMeta_bronze.gif differ diff --git a/docs/static/images/deployingDLTMeta_silver.gif b/docs/static/images/deployingDLTMeta_silver.gif new file mode 100644 index 0000000..aa85b2f Binary files /dev/null and b/docs/static/images/deployingDLTMeta_silver.gif differ diff --git a/docs/static/images/macos_1_databrickslabsmac_installdatabricks.gif b/docs/static/images/macos_1_databrickslabsmac_installdatabricks.gif new file mode 100644 index 0000000..79c3598 Binary files /dev/null and b/docs/static/images/macos_1_databrickslabsmac_installdatabricks.gif differ diff --git a/docs/static/images/onboardingDLTMeta.gif b/docs/static/images/onboardingDLTMeta.gif new file mode 100644 index 0000000..50ddc8b Binary files /dev/null and b/docs/static/images/onboardingDLTMeta.gif differ diff --git a/docs/static/images/onboardingDLTMeta_2.gif b/docs/static/images/onboardingDLTMeta_2.gif new file mode 100644 index 0000000..70156bd Binary files /dev/null and b/docs/static/images/onboardingDLTMeta_2.gif differ diff --git a/docs/static/images/windows_install_databricks.png b/docs/static/images/windows_install_databricks.png new file mode 100644 index 0000000..70cc792 Binary files /dev/null and b/docs/static/images/windows_install_databricks.png differ diff --git a/docs/themes/hugo-theme-learn b/docs/themes/hugo-theme-learn index e817f53..3202533 160000 --- a/docs/themes/hugo-theme-learn +++ b/docs/themes/hugo-theme-learn @@ -1 +1 @@ -Subproject commit e817f53d690d35f181c896e0e320cb40f797e88c +Subproject commit 3202533a746f91c67de1a8fa373c0328ec1b403d diff --git a/labs.yml b/labs.yml index ff41602..bd33b2c 100644 --- a/labs.yml +++ b/labs.yml @@ -4,7 +4,7 @@ description: Metadata-driven framework based on Databricks Delta Live Tables install: script: src/install.py entrypoint: src/cli.py -min_python: 3.10 +min_python: 3.8 commands: - name: onboard description: onboards dlt-meta configuration for bronze and silver layers diff --git a/setup.py b/setup.py index 2d3d334..98e1219 100644 --- a/setup.py +++ b/setup.py @@ -19,7 +19,7 @@ """ setup( name="dlt_meta", - version="0.0.6", + version="0.0.7", python_requires=">=3.8", setup_requires=["wheel>=0.37.1,<=0.42.0"], install_requires=INSTALL_REQUIRES, diff --git a/src/cli.py b/src/cli.py index 9ee251c..05d5604 100644 --- a/src/cli.py +++ b/src/cli.py @@ -5,6 +5,7 @@ import os import sys import uuid +import webbrowser from dataclasses import dataclass from databricks.sdk import WorkspaceClient from databricks.sdk.service import jobs, pipelines, compute @@ -182,6 +183,7 @@ def onboard(self, cmd: OnboardCommand): "launched with run_id={}, Please check the job status in databricks workspace jobs tab" ).format(created_job.job_id, run.run_id) logger.info(msg) + webbrowser.open(f"{self._ws.config.host}/jobs/{created_job.job_id}?o={self._ws.get_workspace_id()}") def create_onnboarding_job(self, cmd: OnboardCommand): """Create the onboarding job.""" @@ -308,6 +310,7 @@ def deploy(self, cmd: DeployCommand): "Please check the pipeline status in databricks workspace under workflows -> Delta Live Tables tab" ) logger.info(msg) + webbrowser.open(f"{self._ws.config.host}/#joblist/pipelines/{pipeline_id}?o={self._ws.get_workspace_id()}/") def _load_onboard_config(self) -> OnboardCommand: onboard_cmd_dict = {}