From 9f24e90d5137459ed462061f4957339575681dc3 Mon Sep 17 00:00:00 2001 From: Amy Chen Date: Mon, 18 Dec 2023 13:24:02 -0500 Subject: [PATCH 1/5] update blog post --- ...023-08-01-announcing-materialized-views.md | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/website/blog/2023-08-01-announcing-materialized-views.md b/website/blog/2023-08-01-announcing-materialized-views.md index 3917e3f192c..e2773361d26 100644 --- a/website/blog/2023-08-01-announcing-materialized-views.md +++ b/website/blog/2023-08-01-announcing-materialized-views.md @@ -11,6 +11,11 @@ hide_table_of_contents: false date: 2023-08-03 is_featured: true --- +:::note +This blog post was updated on December 18, 2023 to cover the support of MVs on dbt-bigquery +and materialized tests. +::: + ## Introduction @@ -32,16 +37,15 @@ Materialized views are now an out of the box materialization in your dbt project - dbt-databricks - dbt-materialize* - dbt-trino* -- dbt-bigquery** +- dbt-bigquery (available on 1.7) *These adapters have supported materialized views in their adapter prior 1.6. -**dbt-bigquery support will be coming in 1.7. Just like you would materialize your sql model as  `table` or `view`  today, you can use `materialized_view` in your model configuration, dbt_project.yml, and resources.yml files. At release, python models will not be supported. -For Postgres/Redshift/Databricks +For Postgres/Redshift/Databricks/Bigquery ```sql {{ @@ -61,6 +65,7 @@ config( }} ``` + :::note We are only supporting dynamic tables on Snowflake, not Snowflake’s materialized views (for a comparison between Snowflake Dynamic Tables and Materialized Views, refer [docs](https://docs.snowflake.com/en/user-guide/dynamic-tables-comparison#dynamic-tables-compared-to-materialized-views). Dynamic tables are better suited for continuous transformations due to functionality like the ability to join, union, and aggregate on base tables, views , and other dynamic tables. Due to those features, they are also more aligned with what other data platforms are calling Materialized Views. For the sake of simplicity, when I refer to materialized views in this blog, I mean dynamic tables in Snowflake. ::: @@ -126,7 +131,7 @@ config( }} ``` -For Redshift: +For Redshift/BigQuery: ```sql {{ @@ -171,12 +176,12 @@ Now if you do need to more fully build out your development pipeline (making sur ### Testing -Now let’s dive into the second question: how do you test? In development and QA, this will look the same as any batch run tests. You can run `dbt build` or  `dbt test` and then have the tests run after execution as validation. But in production, what can you do to continually test? Your options are: +Now let’s dive into the second question: how do you test? In development and QA, this will look the same as any tests you might have while developing your batch pipelines. You can run `dbt build` or  `dbt test` and then have the tests run after execution as validation. But in production, what changes? -- Continue to do batch testing as we wait for [materialized tests](https://github.com/dbt-labs/dbt-core/issues/6914) -- Or overriding the –store-failures macro like what Materialize has created [here](https://materialize.com/blog/real-time-data-quality-tests-using-dbt-and-materialize/) for their adapter to materialize failed rows as a materialized view. This is not a great solution for the long term but if you have urgency to put this into production, it is an option. +I recommend that you update any tests applied to a materialized view/dynamic table with the +[store_failures_as](/reference/resource-configs/store_failures_as) configuration set to true and materialized as a view. This allows you to create a view that will provide all the rows that failed your test at time of query. Please note that this does not provide a historical look. You can also create alerting onto the view if it fails your expectations. -In order to promote materialized views into production, the process will look very much like it did with your incremental models. Using SlimCI, for new MVs, you can build them into your QA environment. For existing MVs without changes, we can skip and defer to the production objects. +In order to promote materialized views into production, the process will look very much like it did with your incremental models. Use [CI jobs](https://docs.getdbt.com/docs/deploy/ci-jobs) with defer so you can build them into your QA environment. For existing MVs without changes, we can skip and defer to the production objects. ### Production From 9ae628c58375a9bd539993d213b56f34098a1315 Mon Sep 17 00:00:00 2001 From: Amy Chen Date: Mon, 18 Dec 2023 13:24:25 -0500 Subject: [PATCH 2/5] update note at top --- website/blog/2023-08-01-announcing-materialized-views.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2023-08-01-announcing-materialized-views.md b/website/blog/2023-08-01-announcing-materialized-views.md index e2773361d26..b8998e4da11 100644 --- a/website/blog/2023-08-01-announcing-materialized-views.md +++ b/website/blog/2023-08-01-announcing-materialized-views.md @@ -13,7 +13,7 @@ is_featured: true --- :::note This blog post was updated on December 18, 2023 to cover the support of MVs on dbt-bigquery -and materialized tests. +and updates on how to test MVs. ::: From c6a95cd52babb4ddb95d378c4f43aaa3728ec7f4 Mon Sep 17 00:00:00 2001 From: Amy Chen Date: Mon, 18 Dec 2023 14:39:36 -0500 Subject: [PATCH 3/5] update the bq config --- .../2023-08-01-announcing-materialized-views.md | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/website/blog/2023-08-01-announcing-materialized-views.md b/website/blog/2023-08-01-announcing-materialized-views.md index b8998e4da11..39a432c26e5 100644 --- a/website/blog/2023-08-01-announcing-materialized-views.md +++ b/website/blog/2023-08-01-announcing-materialized-views.md @@ -131,7 +131,7 @@ config( }} ``` -For Redshift/BigQuery: +For Redshift: ```sql {{ @@ -142,6 +142,18 @@ config( ) }} ``` +For Bigquery +```sql +{{ +config( + materialized = 'materialized_view', + on_configuration_change = 'apply', + auto_refresh = True, + refresh_interval_minutes = 30 + max_staleness = 60, +) +}} +``` For Databricks: From b6f8ea38173155c396400c5ed52402c8ec501615 Mon Sep 17 00:00:00 2001 From: Amy Chen Date: Mon, 18 Dec 2023 14:47:33 -0500 Subject: [PATCH 4/5] update to include links --- .../2023-08-01-announcing-materialized-views.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/website/blog/2023-08-01-announcing-materialized-views.md b/website/blog/2023-08-01-announcing-materialized-views.md index 39a432c26e5..75b11c5a37f 100644 --- a/website/blog/2023-08-01-announcing-materialized-views.md +++ b/website/blog/2023-08-01-announcing-materialized-views.md @@ -31,13 +31,13 @@ Today we are announcing that we now support Materialized Views in dbt. So, what Materialized views are now an out of the box materialization in your dbt project once you upgrade to the latest version of dbt v1.6 on these following adapters: -- dbt-postgres -- dbt-redshift -- dbt-snowflake -- dbt-databricks -- dbt-materialize* -- dbt-trino* -- dbt-bigquery (available on 1.7) +- [dbt-postgres](/reference/resource-configs/postgres-configs#materialized-views) +- [dbt-redshift](reference/resource-configs/redshift-configs#materialized-views) +- [dbt-snowflake](reference/resource-configs/snowflake-configs#dynamic-tables) +- [dbt-databricks](/reference/resource-configs/databricks-configs#materialized-views-and-streaming-tables) +- [dbt-materialize*](/reference/resource-configs/materialize-configs#incremental-models-materialized-views) +- [dbt-trino*](/reference/resource-configs/trino-configs#materialized-view) +- [dbt-bigquery (available on 1.7)](/reference/resource-configs/bigquery-configs#materialized-views) *These adapters have supported materialized views in their adapter prior 1.6. From 593baaeea0c0f2bd743fd23821c97f6f6b424678 Mon Sep 17 00:00:00 2001 From: Amy Chen Date: Mon, 18 Dec 2023 14:48:16 -0500 Subject: [PATCH 5/5] fix the enable_refresh --- website/blog/2023-08-01-announcing-materialized-views.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/blog/2023-08-01-announcing-materialized-views.md b/website/blog/2023-08-01-announcing-materialized-views.md index 75b11c5a37f..eb9716e73a5 100644 --- a/website/blog/2023-08-01-announcing-materialized-views.md +++ b/website/blog/2023-08-01-announcing-materialized-views.md @@ -148,7 +148,7 @@ For Bigquery config( materialized = 'materialized_view', on_configuration_change = 'apply', - auto_refresh = True, + enable_refresh = True, refresh_interval_minutes = 30 max_staleness = 60, )