From 9c103211540363669ddde090486efefb14866e51 Mon Sep 17 00:00:00 2001 From: Camille Kesser <101661315+camillek-db@users.noreply.github.com> Date: Thu, 30 Nov 2023 13:17:35 -0600 Subject: [PATCH 1/7] Update Databricks quickstart --- website/docs/guides/databricks-qs.md | 52 +++++++++++++++++++++++++--- 1 file changed, 47 insertions(+), 5 deletions(-) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index 5a0c5536e7f..a7bc2b0b006 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -167,14 +167,56 @@ If you get a session error and don’t get redirected to this page, you can go b There are two ways to connect dbt Cloud to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt Cloud account from within your new Databricks trial account. The second option is to create your dbt Cloud account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt Cloud setup flow, dbt Labs recommends connecting manually. -If you want to use Partner Connect, refer to [Connect to dbt Cloud using Partner Connect](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-using-partner-connect) in the Databricks docs for instructions. +This quickstart assumes you'll use Partner Connect. -If you want to connect manually, refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. +:::tip + Partner Connect is intended for trial partner accounts. If your organization already has a dbt account, you should connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. +::: + +To connect dbt Cloud to Databricks using Partner Connect, do the following: + +1. In the sidebar, click **Partner Connect**. + +2. Click the dbt tile. + +3. Select a catalog for dbt to write to, and then click **Next**. The drop-down list displays catalogs you own or have access to. If your workspace isn't -enabled, the legacy Hive metastore (`hive_metastore`) is used. + +5. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. + +6. If there are no SQL warehouses in your workspace, do the following: + + a. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. + #. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. + #. Return to the Partner Connect tab in your browser, and then close the dbt tile. + #. Re-open the dbt tile. + #. Select the SQL warehouse you just created from the drop-down list. + +7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you own or have access to. You can repeat this step to add multiple schemas. + + Partner Connect creates the following resources in your workspace: + + - A Databricks service principal named **DBT_CLOUD_USER**. + - A Databricks personal access token that is associated with the **DBT_CLOUD_USER** service principal. + + Partner Connect also grants the following privileges to the **DBT_CLOUD_USER** service principal: + + - (Unity Catalog) **USE CATALOG**: Required to interact with objects within the selected catalog. + - (Unity Catalog) **USE SCHEMA**: Required to interact with objects within the selected schema. + - (Unity Catalog) **CREATE SCHEMA**: Grants the ability to create schemas in the selected catalog. + - (Hive metastore) **USAGE**: Required to grant the **SELECT** and **READ_METADATA** privileges for the schemas you selected. + - **SELECT**: Grants the ability to read the schemas you selected. + - (Hive metastore) **READ_METADATA**: Grants the ability to read metadata for the schemas you selected. + - **CAN_USE**: Grants permissions to use the SQL warehouse you selected. + +8. Click **Next**. + + The **Email** box displays the email address for your Databricks account. dbt uses this email address to prompt you to create a trial dbt account. + +9. Click **Connect to dbt Cloud**. -## Set up a dbt Cloud managed repository -If you used Partner Connect, you can skip to [initializing your dbt project](#initialize-your-dbt-project-and-start-developing) as the Partner Connect provides you with a managed repository. Otherwise, you will need to create your repository connection. + A new tab opens in your web browser, which displays the dbt website. - +10. Complete the on-screen instructions on the dbt website to create your trial dbt account. ## Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt Cloud: From 7063b067007e3b7607248973c58d05b328f80726 Mon Sep 17 00:00:00 2001 From: Camille Kesser <101661315+camillek-db@users.noreply.github.com> Date: Thu, 30 Nov 2023 13:25:31 -0600 Subject: [PATCH 2/7] sub ordered list numbering --- website/docs/guides/databricks-qs.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index a7bc2b0b006..f10127e552b 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -185,11 +185,11 @@ To connect dbt Cloud to Databricks using Partner Connect, do the following: 6. If there are no SQL warehouses in your workspace, do the following: - a. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. - #. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. - #. Return to the Partner Connect tab in your browser, and then close the dbt tile. - #. Re-open the dbt tile. - #. Select the SQL warehouse you just created from the drop-down list. + 1. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. + 2. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. + 3. Return to the Partner Connect tab in your browser, and then close the dbt tile. + 4. Re-open the dbt tile. + 5. Select the SQL warehouse you just created from the drop-down list. 7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you own or have access to. You can repeat this step to add multiple schemas. From 78357deb3f573921b6db5c4b09a7f1543071375b Mon Sep 17 00:00:00 2001 From: Camille Kesser <101661315+camillek-db@users.noreply.github.com> Date: Mon, 11 Dec 2023 11:56:43 -0600 Subject: [PATCH 3/7] Update databricks-qs.md --- website/docs/guides/databricks-qs.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index f10127e552b..1df059fe90c 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -169,7 +169,7 @@ There are two ways to connect dbt Cloud to Databricks. The first option is Partn This quickstart assumes you'll use Partner Connect. -:::tip +:::note Partner Connect is intended for trial partner accounts. If your organization already has a dbt account, you should connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. ::: @@ -179,7 +179,7 @@ To connect dbt Cloud to Databricks using Partner Connect, do the following: 2. Click the dbt tile. -3. Select a catalog for dbt to write to, and then click **Next**. The drop-down list displays catalogs you own or have access to. If your workspace isn't -enabled, the legacy Hive metastore (`hive_metastore`) is used. +3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't -enabled, the legacy Hive metastore (`hive_metastore`) is used. 5. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. @@ -191,7 +191,7 @@ To connect dbt Cloud to Databricks using Partner Connect, do the following: 4. Re-open the dbt tile. 5. Select the SQL warehouse you just created from the drop-down list. -7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you own or have access to. You can repeat this step to add multiple schemas. +7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you have read and write access to. You can repeat this step to add multiple schemas. Partner Connect creates the following resources in your workspace: From 9f305ab74010c6f580297c1af248d70616d8713f Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 30 Jul 2024 16:10:15 -0400 Subject: [PATCH 4/7] Update databricks-qs.md Editorial changes --- website/docs/guides/databricks-qs.md | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index 821c38f0fcd..3cab7a46445 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -169,28 +169,28 @@ If you get a session error and don’t get redirected to this page, you can go b There are two ways to connect dbt Cloud to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt Cloud account from within your new Databricks trial account. The second option is to create your dbt Cloud account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt Cloud setup flow, dbt Labs recommends connecting manually. -This quickstart assumes you'll use Partner Connect. +## Create a dbt Cloud account from Partner Connect :::note - Partner Connect is intended for trial partner accounts. If your organization already has a dbt account, you should connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. + Partner Connect is intended for trial partner accounts. If your organization already has a dbt Cloud account, connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. ::: To connect dbt Cloud to Databricks using Partner Connect, do the following: -1. In the sidebar, click **Partner Connect**. +1. In the sidebar of your Databricks account, click **Partner Connect**. -2. Click the dbt tile. +2. Click the **dbt tile**. -3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't -enabled, the legacy Hive metastore (`hive_metastore`) is used. +3. Select a catalog from the drop-down list, and then click **Next**. The drop-down list displays catalogs you have read and write access to. If your workspace isn't `-enabled`, the legacy Hive metastore (`hive_metastore`) is used. 5. If there are SQL warehouses in your workspace, select a SQL warehouse from the drop-down list. If your SQL warehouse is stopped, click **Start**. -6. If there are no SQL warehouses in your workspace, do the following: +6. If there are no SQL warehouses in your workspace: 1. Click **Create warehouse**. A new tab opens in your browser that displays the **New SQL Warehouse** page in the Databricks SQL UI. 2. Follow the steps in [Create a SQL warehouse](https://docs.databricks.com/en/sql/admin/create-sql-warehouse.html#create-a-sql-warehouse) in the Databricks docs. - 3. Return to the Partner Connect tab in your browser, and then close the dbt tile. - 4. Re-open the dbt tile. + 3. Return to the Partner Connect tab in your browser, and then close the **dbt tile**. + 4. Re-open the **dbt tile**. 5. Select the SQL warehouse you just created from the drop-down list. 7. Select a schema from the drop-down list, and then click **Add**. The drop-down list displays schemas you have read and write access to. You can repeat this step to add multiple schemas. @@ -212,15 +212,16 @@ To connect dbt Cloud to Databricks using Partner Connect, do the following: 8. Click **Next**. - The **Email** box displays the email address for your Databricks account. dbt uses this email address to prompt you to create a trial dbt account. + The **Email** box displays the email address for your Databricks account. dbt Labs uses this email address to prompt you to create a trial dbt Cloud account. 9. Click **Connect to dbt Cloud**. - A new tab opens in your web browser, which displays the dbt website. + A new tab opens in your web browser, which displays the getdbt.com website. -10. Complete the on-screen instructions on the dbt website to create your trial dbt account. +10. Complete the on-screen instructions on the getdbt.com website to create your trial dbt Cloud account. ## Initialize your dbt project​ and start developing + Now that you have a repository configured, you can initialize your project and start development in dbt Cloud: 1. Click **Start developing in the IDE**. It might take a few minutes for your project to spin up for the first time as it establishes your git connection, clones your repo, and tests the connection to the warehouse. From ed8f2a73f5c79cdf8f61224ab0e8506746970fc5 Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 30 Jul 2024 16:11:10 -0400 Subject: [PATCH 5/7] Update website/docs/guides/databricks-qs.md --- website/docs/guides/databricks-qs.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index 3cab7a46445..18e664a9a52 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -169,7 +169,7 @@ If you get a session error and don’t get redirected to this page, you can go b There are two ways to connect dbt Cloud to Databricks. The first option is Partner Connect, which provides a streamlined setup to create your dbt Cloud account from within your new Databricks trial account. The second option is to create your dbt Cloud account separately and build the Databricks connection yourself (connect manually). If you want to get started quickly, dbt Labs recommends using Partner Connect. If you want to customize your setup from the very beginning and gain familiarity with the dbt Cloud setup flow, dbt Labs recommends connecting manually. -## Create a dbt Cloud account from Partner Connect +## Set up the integration from Partner Connect :::note Partner Connect is intended for trial partner accounts. If your organization already has a dbt Cloud account, connect manually. Refer to [Connect to dbt Cloud manually](https://docs.databricks.com/partners/prep/dbt-cloud.html#connect-to-dbt-cloud-manually) in the Databricks docs for instructions. From dd3cdd0f68c23764ef8adb33c01762403d03fc9a Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 30 Jul 2024 16:31:46 -0400 Subject: [PATCH 6/7] Update databricks-qs.md Adding back additional install instructions --- website/docs/guides/databricks-qs.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index 18e664a9a52..0d156674463 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -234,6 +234,10 @@ Now that you have a repository configured, you can initialize your project and s ``` - In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. +## Set up a dbt Cloud managed repository + + + ## Build your first model You have two options for working with files in the dbt Cloud IDE: From 753553c7b6424e953dd0b60fb8b1696439c78d4e Mon Sep 17 00:00:00 2001 From: Matt Shaver <60105315+matthewshaver@users.noreply.github.com> Date: Tue, 30 Jul 2024 16:32:44 -0400 Subject: [PATCH 7/7] Update databricks-qs.md Whoops, wrong order --- website/docs/guides/databricks-qs.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/website/docs/guides/databricks-qs.md b/website/docs/guides/databricks-qs.md index 0d156674463..bb248e09320 100644 --- a/website/docs/guides/databricks-qs.md +++ b/website/docs/guides/databricks-qs.md @@ -220,6 +220,10 @@ To connect dbt Cloud to Databricks using Partner Connect, do the following: 10. Complete the on-screen instructions on the getdbt.com website to create your trial dbt Cloud account. +## Set up a dbt Cloud managed repository + + + ## Initialize your dbt project​ and start developing Now that you have a repository configured, you can initialize your project and start development in dbt Cloud: @@ -234,10 +238,6 @@ Now that you have a repository configured, you can initialize your project and s ``` - In the command line bar at the bottom, enter `dbt run` and click **Enter**. You should see a `dbt run succeeded` message. -## Set up a dbt Cloud managed repository - - - ## Build your first model You have two options for working with files in the dbt Cloud IDE: