Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modified readme and doc site pages #87

Merged
merged 1 commit into from
Aug 8, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ In practice, a single generic DLT pipeline reads the Dataflowspec and uses it to

#### Metadata Interface

- Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json)
- Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.template)
- Capture [Data Quality Rules](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe/customers/bronze_data_quality_expectations.json)
- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json)

Expand Down
2 changes: 1 addition & 1 deletion docs/content/_index.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ In practice, a single generic DLT pipeline reads the Dataflowspec and uses it to
### DLT-META components:

#### Metadata Interface
- Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json)
- Capture input/output metadata in [onboarding file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.template)
- Capture [Data Quality Rules](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe/customers/bronze_data_quality_expectations.json)
- Capture processing logic as sql in [Silver transformation file](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json)

Expand Down
5 changes: 5 additions & 0 deletions docs/content/demo/Append_FLOW_CF.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@ This demo will perform following tasks:
1. Launch Terminal/Command prompt

2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
- Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

```commandline
databricks auth login --host WORKSPACE_HOST
```

3. ```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
Expand Down
5 changes: 5 additions & 0 deletions docs/content/demo/Append_FLOW_EH.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ draft: false
1. Launch Terminal/Command prompt

2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
- Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

```commandline
databricks auth login --host WORKSPACE_HOST
```

3. ```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
Expand Down
18 changes: 15 additions & 3 deletions docs/content/demo/DAIS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,26 @@ This demo showcases DLT-META's capabilities of creating Bronze and Silver DLT pi
1. Launch Terminal/Command promt

2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
- Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

```commandline
databricks auth login --host WORKSPACE_HOST
```

3. ```git clone https://github.com/databrickslabs/dlt-meta.git ```
3. ```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
```

4. ```cd dlt-meta```
4. ```commandline
cd dlt-meta
```

5. Set python environment variable into terminal
```commandline
dlt_meta_home=$(pwd)
```
export PYTHONPATH=<<local dlt-meta path>>
```commandline
export PYTHONPATH=$dlt_meta_home
```

6. Run the command ```python demo/launch_dais_demo.py --username=<<your databricks username>> --source=cloudfiles --uc_catalog_name=<<uc catalog name>> --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/dais-dlt-meta-demo-automated_new```
Expand Down
7 changes: 6 additions & 1 deletion docs/content/demo/Silver_Fanout.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,12 @@ draft: false
1. Launch Terminal/Command prompt

2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)

- Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

```commandline
databricks auth login --host WORKSPACE_HOST
```

3. ```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
```
Expand Down
18 changes: 15 additions & 3 deletions docs/content/demo/Techsummit.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,26 @@ This demo will launch auto generated tables(100s) inside single bronze and silve
1. Launch Terminal/Command prompt

2. Install [Databricks CLI](https://docs.databricks.com/dev-tools/cli/index.html)
- Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

```commandline
databricks auth login --host WORKSPACE_HOST
```

3. ```git clone https://github.com/databrickslabs/dlt-meta.git ```
3. ```commandline
git clone https://github.com/databrickslabs/dlt-meta.git
```

4. ```cd dlt-meta```
4. ```commandline
cd dlt-meta
```

5. Set python environment variable into terminal
```commandline
dlt_meta_home=$(pwd)
```
export PYTHONPATH=<<local dlt-meta path>>
```commandline
export PYTHONPATH=$dlt_meta_home
```

6. Run the command ```python demo/launch_techsummit_demo.py [email protected] --source=cloudfiles --cloud_provider_name=aws --dbr_version=13.3.x-scala2.12 --dbfs_path=dbfs:/techsummit-dlt-meta-demo-automated ```
Expand Down
5 changes: 5 additions & 0 deletions docs/content/getting_started/dltmeta_cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,11 @@ draft: false

### pre-requisites:
- [Databricks CLI](https://docs.databricks.com/en/dev-tools/cli/tutorial.html)
- Once you install Databricks CLI, authenticate your current machine to a Databricks Workspace:

```commandline
databricks auth login --host WORKSPACE_HOST
```
- Python 3.8.0 +
##### Steps:
1. ``` git clone https://github.com/databrickslabs/dlt-meta.git ```
Expand Down
Loading