-
Notifications
You must be signed in to change notification settings - Fork 66
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #45 from databrickslabs/feature/issue_41
Feature/issue 41
- Loading branch information
Showing
20 changed files
with
135 additions
and
194 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -115,6 +115,7 @@ celerybeat.pid | |
# Environments | ||
.env | ||
.venv | ||
.venvclear/ | ||
env/ | ||
venv/ | ||
ENV/ | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -3,7 +3,8 @@ | |
2. [Databricks Techsummit Demo](#databricks-tech-summit-fy2024-demo): 100s of data sources ingestion in bronze and silver DLT pipelines automatically. | ||
|
||
|
||
# DAIS 2023 DEMO | ||
# DAIS 2023 DEMO | ||
## [DAIS 2023 Session Recording](https://www.youtube.com/watch?v=WYv5haxLlfA) | ||
This Demo launches Bronze and Silver DLT pipleines with following activities: | ||
- Customer and Transactions feeds for initial load | ||
- Adds new feeds Product and Stores to existing Bronze and Silver DLT pipelines with metadata changes. | ||
|
@@ -23,7 +24,7 @@ This Demo launches Bronze and Silver DLT pipleines with following activities: | |
export PYTHONPATH=<<local dlt-meta path>> | ||
``` | ||
|
||
6. Run the command ```python demo/launch_dais_demo.py --username=<<your databricks username>> --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-demo-automated_new``` | ||
6. Run the command ```python demo/launch_dais_demo.py --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-demo-automated_new``` | ||
- cloud_provider_name : aws or azure or gcp | ||
- db_version : Databricks Runtime Version | ||
- dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines | ||
|
@@ -61,7 +62,7 @@ This demo will launch auto generated tables(100s) inside single bronze and silve | |
export PYTHONPATH=<<local dlt-meta path>> | ||
``` | ||
|
||
6. Run the command ```python demo/launch_techsummit_demo.py --[email protected] --source=cloudfiles --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/techsummit-dlt-meta-demo-automated ``` | ||
6. Run the command ```python demo/launch_techsummit_demo.py --source=cloudfiles --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/techsummit-dlt-meta-demo-automated ``` | ||
- cloud_provider_name : aws or azure or gcp | ||
- db_version : Databricks Runtime Version | ||
- dbfs_path : Path on your Databricks workspace where demo will be copied for launching DLT-META Pipelines | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.