Merge branch 'current' into dbeatty/target-path-for-artifacts

website/blog/2021-11-23-how-to-upgrade-dbt-versions.md

@@ -12,6 +12,18 @@ date: 2021-11-29
 is_featured: true
 ---
 
+:::tip February 2024 Update
+
+It's been a few years since dbt-core turned 1.0! Since then, we've committed to releasing zero breaking changes whenever possible and it's become much easier to upgrade dbt Core versions.
+
+In 2024, we're taking this promise further by:
+- Stabilizing interfaces for everyone — adapter maintainers, metadata consumers, and (of course) people writing dbt code everywhere — as discussed in [our November 2023 roadmap update](https://github.com/dbt-labs/dbt-core/blob/main/docs/roadmap/2023-11-dbt-tng.md).
+- Introducing **Keep on latest version** in dbt Cloud. No more manual upgrades and no more need for _a second sandbox project_ just to try out new features in development. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud).
+
+We're leaving the rest of this post as is, so we can all remember how it used to be. Enjoy a stroll down memory lane.
+
+:::
+
 As we get closer to dbt v1.0 shipping in December, it's a perfect time to get your installation up to scratch. dbt 1.0 represents the culmination of over five years of development and refinement to the analytics engineering experience - smoothing off sharp edges, speeding up workflows and enabling whole new classes of work.
 
 Even with all the new shinies on offer, upgrading can be daunting – you rely on dbt to power your analytics workflow and can’t afford to change things just to discover that your daily run doesn’t work anymore. I’ve been there. This is the checklist I wish I had when I owned my last company’s dbt project.

website/docs/best-practices/clone-incremental-models.md

@@ -7,7 +7,7 @@ hoverSnippet: Learn how to clone incremental models for CI jobs.
 ---
 
 Before you begin, you must be aware of a few conditions:
-- `dbt clone` is only available with dbt version 1.6 and newer. Refer to our [upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud) for help enabling newer versions in dbt Cloud
+- `dbt clone` is only available with dbt version 1.6 and newer. Refer to our [upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud) for help enabling newer versions in dbt Cloud
 - This strategy only works for warehouse that support zero copy cloning (otherwise `dbt clone` will just create pointer views).
 - Some teams may want to test that their incremental models run in both incremental mode and full-refresh mode.
 

website/docs/best-practices/how-we-mesh/mesh-4-faqs.md

@@ -273,7 +273,7 @@ The [dbt Cloud CLI](/docs/cloud/cloud-cli-installation) allows users to develop
 
 <detailsToggle alt_header="Does dbt Mesh require me to be on a specific version of dbt?">
 
-Yes, your account must be on [at least dbt v1.6](/docs/dbt-versions/upgrade-core-in-cloud) to take advantage of [cross-project dependencies](/docs/collaborate/govern/project-dependencies), one of the most crucial underlying capabilities required to implement a dbt Mesh.
+Yes, your account must be on [at least dbt v1.6](/docs/dbt-versions/upgrade-dbt-version-in-cloud) to take advantage of [cross-project dependencies](/docs/collaborate/govern/project-dependencies), one of the most crucial underlying capabilities required to implement a dbt Mesh.
 
 </detailsToggle>
 

website/docs/docs/build/about-metricflow.md

@@ -15,7 +15,7 @@ MetricFlow handles SQL query construction and defines the specification for dbt
 Before you start, consider the following guidelines:
 
 - Define metrics in YAML and query them using these [new metric specifications](https://github.com/dbt-labs/dbt-core/discussions/7456).
-- You must be on [dbt version](/docs/dbt-versions/upgrade-core-in-cloud) 1.6 or higher to use MetricFlow. 
+- You must be on [dbt version](/docs/dbt-versions/upgrade-dbt-version-in-cloud) 1.6 or higher to use MetricFlow. 
 - Use MetricFlow with Snowflake, BigQuery, Databricks, Postgres (dbt Core only), or Redshift. 
 - Discover insights and query your metrics using the [dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) and its diverse range of [available integrations](/docs/use-dbt-semantic-layer/avail-sl-integrations). 
 

website/docs/docs/build/unit-tests.md

@@ -0,0 +1,258 @@
+---
+title: "Unit tests"
+sidebar_label: "Unit tests"
+description: "Learn how to use unit tests on your SQL models."
+search_weight: "heavy"
+id: "unit-tests"
+keywords:
+  - unit test, unit tests, unit testing, dag
+---
+:::note closed beta
+
+Support for unit testing dbt models is available to dbt Cloud customers who have chosen to [Keep on latest version](/docs/dbt-versions/upgrade-dbt-version-in-cloud#keep-on-latest-version). This is currently in closed beta, starting with select customers and rolling out to wider availability through February and March.
+
+For dbt Core, unit tests will be available in v1.8, planned for late April 2024. The v1.8.0-b1 release of `dbt-core` & dbt Labs-maintained adapters is planned for February 28.
+
+:::
+
+Historically, dbt's test coverage was confined to [“data” tests](/docs/build/data-tests), assessing the quality of input data or resulting datasets' structure. However, these tests could only be executed _after_ a building a model. 
+
+Now, we are introducing a new type of test to dbt - unit tests. In software programming, unit tests validate small portions of your functional code, and they work much the same way here. Unit tests allow you to validate your SQL modeling logic on a small set of static inputs _before_ you materialize your full model in production. Unit tests enable test-driven development, benefiting developer efficiency and code reliability. 
+
+## Before you begin
+
+- We currently only support unit testing SQL models.
+- We currently only support adding unit tests to models in your _current_ project.
+- If your model has multiple versions, by default the unit test will run on *all* versions of your model. Read [unit testing versioned models](#unit-testing-versioned-models) for more information.
+
+Read the [reference doc](/reference/resource-properties/unit-tests) for more details about formatting your unit tests.
+
+### When to add a unit test to your model
+
+You should unit test a model:
+- When your SQL contains complex logic:
+    - Regex
+    - Date math
+    - Window functions
+    - `case when` statements when there are many `when`s
+    - Truncation
+    - Recursion
+- When you're writing custom logic to process input data, similar to creating a function.
+- We don't recommend conducting unit testing for functions like `min()` since these functions are tested extensively by the warehouse. If an unexpected issue arises, it's more likely a result of issues in the underlying data rather than the function itself. Therefore, fixture data in the unit test won't provide valuable information.
+- Logic for which you had bugs reported before.
+- Edge cases not yet seen in your actual data that you want to handle.
+- Prior to refactoring the transformation logic (especially if the refactor is significant).
+- Models with high "criticality" (public, contracted models or models directly upstream of an exposure).
+
+## Unit testing a model
+
+This example creates a new `dim_customers` model with a field `is_valid_email_address` that calculates whether or not the customer’s email is valid: 
+
+<file name='dim_customers.sql'>
+
+```sql
+with customers as (
+
+    select * from {{ ref('stg_customers') }}
+
+),
+
+accepted_email_domains as (
+
+    select * from {{ ref('top_level_email_domains') }}
+
+),
+
+check_valid_emails as (
+
+    select
+        customers.customer_id,
+        customers.first_name,
+        customers.last_name,
+	coalesce (regexp_like(
+            customers.email, '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\\.[A-Za-z]{2,}$'
+        )
+        = true
+        and accepted_email_domains.tld is not null,
+        false) as is_valid_email_address
+    from customers
+		left join accepted_email_domains
+        on customers.email_top_level_domain = lower(accepted_email_domains.tld)
+
+)
+
+select * from check_valid_emails
+```
+</file>
+
+The logic posed in this example can be challenging to validate. You can add a unit test to this model to ensure the `is_valid_email_address` logic captures all known edge cases: emails without `.`, emails without `@`, and emails from invalid domains.
+
+<file name='dbt_project.yml'> 
+
+```yaml
+unit_tests:
+  - name: test_is_valid_email_address
+    description: "Check my is_valid_email_address logic captures all known edge cases - emails without ., emails without @, and emails from invalid domains."
+    model: dim_customers
+    given:
+      - input: ref('stg_customers')
+        rows:
+          - {customer_id: 1, email: [email protected], email_top_level_domain: example.com}
+          - {customer_id: 2, email: [email protected], email_top_level_domain: unknown.com}
+          - {customer_id: 3, email: badgmail.com, email_top_level_domain: gmail.com}
+          - {customer_id: 4, email: missingdot@gmailcom, email_top_level_domain: gmail.com}
+      - input: ref('top_level_email_domains')
+        rows:
+          - {tld: example.com}
+          - {tld: gmail.com}
+    expect:
+      rows:
+        - {customer_id: 1, is_valid_email_address: true}
+        - {customer_id: 2, is_valid_email_address: false}
+        - {customer_id: 3, is_valid_email_address: false}
+        - {customer_id: 4, is_valid_email_address: false}
+
+```
+</file>
+
+The previous example defines the mock data using the inline `dict` format, but you can also use `csv` either inline or in a separate fixture file. 
+
+You only have to define the mock data for the columns you care about. This enables you to write succinct and _specific_ unit tests.
+
+:::note
+
+The direct parents of the model that you’re unit testing (in this example, `stg_customers` and `top_level_email_domains`) need to exist in the warehouse before you can execute the unit test.
+
+Use the `--empty` flag to build an empty version of the models to save warehouse spend. 
+
+```bash
+
+dbt run --select "stg_customers top_level_email_domains" --empty
+
+```
+
+Alternatively, use `dbt build` to, in lineage order:
+
+- Run the unit tests on your model.
+- Materialize your model in the warehouse.
+- Run the data tests on your model.
+
+:::
+
+Now you’re ready to run this unit test. You have a couple of options for commands depending on how specific you want to be: 
+
+- `dbt test --select dim_customers` runs _all_ of the tests on `dim_customers`.
+- `dbt test --select "dim_customers,test_type:unit"` runs all of the _unit_ tests on `dim_customers`.
+- `dbt test --select test_is_valid_email_address` runs the test named `test_is_valid_email_address`.
+
+```shell
+
+dbt test --select test_is_valid_email_address
+16:03:49  Running with dbt=1.8.0-a1
+16:03:49  Registered adapter: postgres=1.8.0-a1
+16:03:50  Found 6 models, 5 seeds, 4 data tests, 0 sources, 0 exposures, 0 metrics, 410 macros, 0 groups, 0 semantic models, 1 unit test
+16:03:50  
+16:03:50  Concurrency: 5 threads (target='postgres')
+16:03:50  
+16:03:50  1 of 1 START unit_test dim_customers::test_is_valid_email_address ................... [RUN]
+16:03:51  1 of 1 FAIL 1 dim_customers::test_is_valid_email_address ............................ [FAIL 1 in 0.26s]
+16:03:51  
+16:03:51  Finished running 1 unit_test in 0 hours 0 minutes and 0.67 seconds (0.67s).
+16:03:51  
+16:03:51  Completed with 1 error and 0 warnings:
+16:03:51  
+16:03:51  Failure in unit_test test_is_valid_email_address (models/marts/unit_tests.yml)
+16:03:51    
+
+actual differs from expected:
+
+@@ ,customer_id,is_valid_email_address
+→  ,1        ,True→False
+   ,2        ,False
+...,...      ,...
+
+
+16:03:51  
+16:03:51    compiled Code at models/marts/unit_tests.yml
+16:03:51  
+16:03:51  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
+
+```
+
+The clever regex statement wasn’t as clever as initially thought, as the model incorrectly flagged `[email protected]` (customer 1's email) as an invalid email address.
+
+Updating the regex logic to `'^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$'` (those pesky escape characters) and rerunning the unit test solves the problem:
+
+```shell
+
+dbt test --select test_is_valid_email_address
+16:09:11  Running with dbt=1.8.0-a1
+16:09:12  Registered adapter: postgres=1.8.0-a1
+16:09:12  Found 6 models, 5 seeds, 4 data tests, 0 sources, 0 exposures, 0 metrics, 410 macros, 0 groups, 0 semantic models, 1 unit test
+16:09:12  
+16:09:13  Concurrency: 5 threads (target='postgres')
+16:09:13  
+16:09:13  1 of 1 START unit_test dim_customers::test_is_valid_email_address ................... [RUN]
+16:09:13  1 of 1 PASS dim_customers::test_is_valid_email_address .............................. [PASS in 0.26s]
+16:09:13  
+16:09:13  Finished running 1 unit_test in 0 hours 0 minutes and 0.75 seconds (0.75s).
+16:09:13  
+16:09:13  Completed successfully
+16:09:13  
+16:09:13  Done. PASS=1 WARN=0 ERROR=0 SKIP=0 TOTAL=1
+
+```
+
+Your model is now ready for production! Adding this unit test helped catch an issue with the SQL logic _before_ you materialized `dim_customers` in your warehouse and will better ensure the reliability of this model in the future. 
+
+## Unit testing versioned models
+
+When a unit test is added to a model, it will run on all versions of the model by default.
+Using the example in this article, if you have versions 1, 2, and 3 of `dim_customers`, the `test_is_valid_email_address` unit test will run on all 3 versions.
+
+To only unit test a specific version (or versions) of a model, include the desired version(s) in the model config:
+
+```yml
+
+unit_tests::
+  - name: test_is_valid_email_address
+    model: dim_customers
+      versions:
+        include: 
+          - 2
+    ...
+
+```
+
+In this scenario, if you have version 1, 2, and 3 of `dim_customers `, my `test_is_valid_email_address` unit test will run on _only_ version 2.
+
+To unit test all versions except a specific version (or versions) of a model, you can exclude the relevant version(s) in the model config:
+
+```yml
+
+unit_tests:
+  - name: test_is_valid_email_address
+    model: dim_customers
+      versions:
+        exclude: 
+          - 1
+    ...
+
+```
+So, if you have versions 1, 2, and 3 of `dim_customers`, your `test_is_valid_email_address` unit test will run on _only_ versions 2 and 3.
+
+If you want to unit test a model that references the pinned version of the model, you should specify that in the `ref` of your input:
+
+```yml
+
+unit_tests:
+  - name: test_is_valid_email_address
+    model: dim_customers
+    given: 
+      - input: ref('stg_customers', v=1)
+      ...
+
+```
+
+
+

website/docs/docs/cloud/cloud-cli-installation.md

@@ -25,7 +25,7 @@ dbt commands are run against dbt Cloud's infrastructure and benefit from:
 ## Prerequisites 
 The dbt Cloud CLI is available in all [deployment regions](/docs/cloud/about-cloud/access-regions-ip-addresses) and for both multi-tenant and single-tenant accounts (Azure single-tenant not supported at this time).
 
-- Ensure you are using dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-core-in-cloud) to upgrade.
+- Ensure you are using dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) to upgrade.
 - Note that SSH tunneling for [Postgres and Redshift](/docs/cloud/connect-data-platform/connect-redshift-postgresql-alloydb) connections doesn't support the dbt Cloud CLI yet.
 
 ## Install dbt Cloud CLI

website/docs/docs/cloud/configure-cloud-cli.md

@@ -15,7 +15,7 @@ import CloudCLIFlag from '/snippets/_cloud-cli-flag.md';
 - You must set up a project in dbt Cloud.
   - **Note** — If you're using the dbt Cloud CLI, you can connect to your [data platform](/docs/cloud/connect-data-platform/about-connections) directly in the dbt Cloud interface and don't need a [`profiles.yml`](/docs/core/connect-data-platform/profiles.yml) file. 
 - You must have your [personal development credentials](/docs/dbt-cloud-environments#set-developer-credentials) set for that project. The dbt Cloud CLI will use these credentials, stored securely in dbt Cloud, to communicate with your data platform.
-- You must be on dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-core-in-cloud) to upgrade.
+- You must be on dbt version 1.5 or higher. Refer to [dbt Cloud versions](/docs/dbt-versions/upgrade-dbt-version-in-cloud) to upgrade.
 
 ## Configure the dbt Cloud CLI
 

website/docs/docs/dbt-versions/core-upgrade/01-upgrading-to-v1.8.md

@@ -1,5 +1,5 @@
 ---
-title: "Upgrading to v1.8 (alpha)"
+title: "Upgrading to v1.8 (beta)"
 id: upgrading-to-v1.8
 description: New features and changes in dbt Core v1.8
 displayed_sidebar: "docs"
@@ -9,9 +9,19 @@ displayed_sidebar: "docs"
 
 - Changelog (coming soon)
 - [CLI Installation guide](/docs/core/installation-overview)
-- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
 - Release schedule (coming soon)
 
+:::tip Keep on latest version, always
+
+Starting this year, let dbt Labs handle version upgrades for you. With dbt Cloud, you can get early access to new functionality before it lands in the final release of dbt Core v1.8 and without the need of managing your own version upgrades. For more details, refer to [Upgrade Core version in Cloud](/docs/dbt-versions/upgrade-dbt-version-in-cloud).
+
+The **Keep on latest version** setting is currently available in beta for select dbt Cloud customers, rolling out to wider availability through February and March.
+
+For users of dbt Core, the v1.8.0-b1 release of `dbt-core` & dbt Labs-maintained adapters is planned for February 28.
+
+:::
+
 ## What to know before upgrading
 
 dbt Labs is committed to providing backward compatibility for all versions 1.x, with the exception of any changes explicitly mentioned below. If you encounter an error upon upgrading, please let us know by [opening an issue](https://github.com/dbt-labs/dbt-core/issues/new).

website/docs/docs/dbt-versions/core-upgrade/02-upgrading-to-v1.7.md

@@ -9,7 +9,7 @@ displayed_sidebar: "docs"
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/8aaed0e29f9560bc53d9d3e88325a9597318e375/CHANGELOG.md)
 - [CLI Installation guide](/docs/core/installation-overview)
-- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
 - [Release schedule](https://github.com/dbt-labs/dbt-core/issues/8260)
 
 ## What to know before upgrading

website/docs/docs/dbt-versions/core-upgrade/03-upgrading-to-v1.6.md

@@ -14,7 +14,7 @@ dbt Core v1.6 has three significant areas of focus:
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.6.latest/CHANGELOG.md)
 - [dbt Core installation guide](/docs/core/installation-overview)
-- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
 - [Release schedule](https://github.com/dbt-labs/dbt-core/issues/7481)
 
 ## What to know before upgrading

website/docs/docs/dbt-versions/core-upgrade/05-upgrading-to-v1.5.md

@@ -13,7 +13,7 @@ dbt Core v1.5 is a feature release, with two significant additions:
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.5.latest/CHANGELOG.md)
 - [CLI Installation guide](/docs/core/installation-overview)
-- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
 - [Release schedule](https://github.com/dbt-labs/dbt-core/issues/6715)
 
 ## What to know before upgrading

website/docs/docs/dbt-versions/core-upgrade/06-upgrading-to-v1.4.md

@@ -9,7 +9,7 @@ displayed_sidebar: "docs"
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.4.latest/CHANGELOG.md)
 - [CLI Installation guide](/docs/core/installation-overview)
-- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
 
 **Final release:** January 25, 2023
 

website/docs/docs/dbt-versions/core-upgrade/07-upgrading-to-v1.3.md

@@ -9,7 +9,7 @@ displayed_sidebar: "docs"
 
 - [Changelog](https://github.com/dbt-labs/dbt-core/blob/1.3.latest/CHANGELOG.md)
 - [CLI Installation guide](/docs/core/installation-overview)
-- [Cloud upgrade guide](/docs/dbt-versions/upgrade-core-in-cloud)
+- [Cloud upgrade guide](/docs/dbt-versions/upgrade-dbt-version-in-cloud)
 
 ## What to know before upgrading