From 9910639751162f1b66cc1c7b2bf5929cb072a096 Mon Sep 17 00:00:00 2001 From: Ben Cassell <98852248+benc-db@users.noreply.github.com> Date: Thu, 5 Oct 2023 08:57:42 -0700 Subject: [PATCH 1/2] Update databricks-configs.md Updating to call out incompatibility of insert_overwrite with SQL Warehouses --- website/docs/reference/resource-configs/databricks-configs.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index e57e1efc04a..9bfe4d862cf 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -182,6 +182,9 @@ insert overwrite table analytics.databricks_incremental +#### Usage Notes +* This strategy is not currently compatible with SQL Warehouses + ### The `merge` strategy The `merge` incremental strategy requires: From 19373134c747deb236f5f42c2a689a822936a0b2 Mon Sep 17 00:00:00 2001 From: Ben Cassell <98852248+benc-db@users.noreply.github.com> Date: Wed, 29 Nov 2023 13:07:11 -0800 Subject: [PATCH 2/2] Update databricks-configs.md Incorporating suggestions --- .../docs/reference/resource-configs/databricks-configs.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/website/docs/reference/resource-configs/databricks-configs.md b/website/docs/reference/resource-configs/databricks-configs.md index 9bfe4d862cf..0dc3f8c5616 100644 --- a/website/docs/reference/resource-configs/databricks-configs.md +++ b/website/docs/reference/resource-configs/databricks-configs.md @@ -100,6 +100,10 @@ insert into table analytics.databricks_incremental ### The `insert_overwrite` strategy +:::caution +This strategy is currently only compatible with All Purpose Clusters, not SQL Warehouses. +::: + This strategy is most effective when specified alongside a `partition_by` clause in your model config. dbt will run an [atomic `insert overwrite` statement](https://spark.apache.org/docs/3.0.0-preview/sql-ref-syntax-dml-insert-overwrite-table.html) that dynamically replaces all partitions included in your query. Be sure to re-select _all_ of the relevant data for a partition when using this incremental strategy. If no `partition_by` is specified, then the `insert_overwrite` strategy will atomically replace all contents of the table, overriding all existing data with only the new records. The column schema of the table remains the same, however. This can be desirable in some limited circumstances, since it minimizes downtime while the table contents are overwritten. The operation is comparable to running `truncate` + `insert` on other databases. For atomic replacement of Delta-formatted tables, use the `table` materialization (which runs `create or replace`) instead. @@ -182,9 +186,6 @@ insert overwrite table analytics.databricks_incremental -#### Usage Notes -* This strategy is not currently compatible with SQL Warehouses - ### The `merge` strategy The `merge` incremental strategy requires: