Skip to content

Commit

Permalink
Corrected onboarding files to templates and removed full paths
Browse files Browse the repository at this point in the history
  • Loading branch information
ravi-databricks committed Jul 22, 2024
1 parent 774742f commit 545dbd7
Show file tree
Hide file tree
Showing 5 changed files with 15 additions and 16 deletions.
8 changes: 4 additions & 4 deletions docs/content/faq/execution.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Please refer to the [Getting Started]({{%relref "getting_started/_index.md" %}})
**Q. How do I create metadata DLT-META ?**

DLT-META needs following metadata files:
- [Onboarding File](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.json) captures input/output metadata
- [Onboarding File](https://github.com/databrickslabs/dlt-meta/blob/main/examples/onboarding.template) captures input/output metadata
- [Data Quality Rules File](https://github.com/databrickslabs/dlt-meta/tree/main/examples/dqe) captures data quality rules
- [Silver transformation File](https://github.com/databrickslabs/dlt-meta/blob/main/examples/silver_transformations.json) captures processing logic as sql

Expand All @@ -28,7 +28,7 @@ DLT-META uses data_flow_group to launch DLT pipelines, so all the tables belongs
**Q. Can we run onboarding for bronze layer only?**

Yes! Please follow below steps:
1. Bronze Metadata preparation ([example](https://github.com/databrickslabs/dlt-meta/blob/main/integration_tests/conf/cloudfiles-onboarding_A2.template))
1. Bronze Metadata preparation ([example](https://github.com/databrickslabs/dlt-meta/blob/main/examples/bronze_onboarding.template))
2. Onboarding Job
- Option#1: [DLT-META CLI](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_cli/#onboardjob)
- Option#2: [Manual Job](https://databrickslabs.github.io/dlt-meta/getting_started/dltmeta_manual/#onboardjob)
Expand All @@ -37,7 +37,7 @@ Yes! Please follow below steps:
{
"onboard_layer": "bronze",
"database": "dlt_demo",
"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
"import_author": "Ravi",
"version": "v1",
Expand All @@ -50,7 +50,7 @@ Yes! Please follow below steps:
```
onboarding_params_map = {
"database": "uc_name.dlt_demo",
"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
"overwrite": "True",
"env": "dev",
Expand Down
8 changes: 4 additions & 4 deletions docs/content/getting_started/dltmeta_manual.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ draft: false
{
"onboard_layer": "bronze_silver",
"database": "dlt_demo",
"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
"silver_dataflowspec_table": "silver_dataflowspec_table",
"silver_dataflowspec_path": "dbfs:/onboarding_tables_cdc/silver",
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
Expand All @@ -51,7 +51,7 @@ draft: false
{
"onboard_layer": "bronze_silver",
"database": "uc_name.dlt_demo",
"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
"silver_dataflowspec_table": "silver_dataflowspec_table",
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
"import_author": "Ravi",
Expand All @@ -78,7 +78,7 @@ Alternatly you can enter keyword arguments, click + Add and enter a key and valu
```python
onboarding_params_map = {
"database": "dlt_demo",
"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
"bronze_dataflowspec_path": "dbfs:/onboarding_tables_cdc/bronze",
"silver_dataflowspec_table": "silver_dataflowspec_table",
Expand All @@ -96,7 +96,7 @@ OnboardDataflowspec(spark, onboarding_params_map).onboard_dataflow_specs()
```python
onboarding_params_map = {
"database": "uc_name.dlt_demo",
"onboarding_file_path": "dbfs:/onboarding_files/users_onboarding.json",
"onboarding_file_path": "dbfs:/dlt-meta/conf/onboarding.json",,
"bronze_dataflowspec_table": "bronze_dataflowspec_table",
"silver_dataflowspec_table": "silver_dataflowspec_table",
"overwrite": "True",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
"source_details": {
"source_database": "APP",
"source_table": "CUSTOMERS",
"source_path_it": "dbfs:/tmp/DLT-META//eedf48c2a67647f2b438ba4a3a757e7a/integration_tests/resources/data/customers_delta",
"source_path_it": "{dbfs_path}/resources/data/customers_delta",
"source_metadata": {
"include_autoloader_metadata_column": "True",
"autoloader_metadata_col_name": "source_metadata",
Expand All @@ -16,24 +16,24 @@
"input_file_path": "_metadata.file_path"
}
},
"source_schema_path": "dbfs:/tmp/DLT-META//eedf48c2a67647f2b438ba4a3a757e7a/integration_tests/resources/customers.ddl"
"source_schema_path": "{dbfs_path}/resources/customers.ddl"
},
"bronze_database_it": "ravi_dlt_meta_uc.dlt_meta_bronze_it_eedf48c2a67647f2b438ba4a3a757e7a",
"bronze_database_it": "{uc_catalog_name}.{bronze_schema}",
"bronze_table": "customers_delta",
"bronze_reader_options": {
"cloudFiles.format": "json",
"cloudFiles.inferColumnTypes": "true",
"cloudFiles.rescuedDataColumn": "_rescued_data"
},
"bronze_table_path_it": "dbfs:/tmp/DLT-META//eedf48c2a67647f2b438ba4a3a757e7a/data/bronze/customers_delta",
"bronze_table_path_it": "{dbfs_path}data/bronze/customers_delta",
"bronze_table_properties": {
"pipelines.autoOptimize.managed": "true",
"pipelines.autoOptimize.zOrderCols": "id, email"
},
"bronze_data_quality_expectations_json_it": "dbfs:/tmp/DLT-META//eedf48c2a67647f2b438ba4a3a757e7a/integration_tests/conf/dqe/customers/bronze_data_quality_expectations.json",
"bronze_database_quarantine_it": "ravi_dlt_meta_uc.dlt_meta_bronze_it_eedf48c2a67647f2b438ba4a3a757e7a",
"bronze_data_quality_expectations_json_it": "{dbfs_path}/conf/dqe/customers/bronze_data_quality_expectations.json",
"bronze_database_quarantine_it": "{uc_catalog_name}.{bronze_schema}",
"bronze_quarantine_table": "customers_delta_quarantine",
"bronze_quarantine_table_path_it": "dbfs:/tmp/DLT-META//eedf48c2a67647f2b438ba4a3a757e7a/data/bronze/customers_quarantine_delta",
"bronze_quarantine_table_path_it": "{dbfs_path}data/bronze/customers_quarantine_delta",
"bronze_quarantine_table_properties": {
"pipelines.reset.allowed": "false",
"pipelines.autoOptimize.zOrderCols": "id, email"
Expand Down
File renamed without changes.
1 change: 0 additions & 1 deletion integration_tests/conf/onboarding_A2.json

This file was deleted.

0 comments on commit 545dbd7

Please sign in to comment.