From 13bab21da6151eec98606817122c406d3e544faa Mon Sep 17 00:00:00 2001 From: Jeremy Cohen Date: Tue, 1 Oct 2024 23:31:08 +0200 Subject: [PATCH] Document new recommended syntax for `foreign_key` constraints (#6189) ## What are you changing in this pull request and why? Update documentation on `constraints` to prominently feature new recommended syntax for defining `foreign_key` constraints. Resolves #5983 ## Checklist - [x] I have reviewed the [Content style guide](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/content-style-guide.md) so my content adheres to these guidelines. - [x] The topic I'm writing about is for specific dbt version(s) and I have versioned it according to the [version a whole page](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#adding-a-new-version) and/or [version a block of content](https://github.com/dbt-labs/docs.getdbt.com/blob/current/contributing/single-sourcing-content.md#versioning-blocks-of-content) guidelines. - [x] I have added checklist item(s) to this list for anything anything that needs to happen before this PR is merged, such as "needs technical review" or "change base branch." --------- Co-authored-by: Natalie Fiann Co-authored-by: Mirna Wong <89008547+mirnawong1@users.noreply.github.com> --- .../docs/docs/dbt-versions/release-notes.md | 1 + .../resource-properties/constraints.md | 82 ++++++++++++++----- 2 files changed, 64 insertions(+), 19 deletions(-) diff --git a/website/docs/docs/dbt-versions/release-notes.md b/website/docs/docs/dbt-versions/release-notes.md index 7c2614b2c10..d152fc92cf7 100644 --- a/website/docs/docs/dbt-versions/release-notes.md +++ b/website/docs/docs/dbt-versions/release-notes.md @@ -20,6 +20,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo ## September 2024 +- **New**: Use the new recommended syntax for [defining `foreign_key` constraints](/reference/resource-properties/constraints) using `refs`, available in dbt Cloud Versionless. This will soon be released in dbt Core v1.9. This new syntax will capture dependencies and works across different environments. - **Enhancement**: You can now run [Semantic Layer commands](/docs/build/metricflow-commands) commands in the [dbt Cloud IDE](/docs/cloud/dbt-cloud-ide/develop-in-the-cloud). The supported commands are `dbt sl list`, `dbt sl list metrics`, `dbt sl list dimension-values`, `dbt sl list saved-queries`, `dbt sl query`, `dbt sl list dimensions`, `dbt sl list entities`, and `dbt sl validate`. - **New**: Microsoft Excel, a dbt Semantic Layer integration, is now generally available. The integration allows you to connect to Microsoft Excel to query metrics and collaborate with your team. Available for [Excel Desktop](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100&rs=en-US&correlationId=4132ecd1-425d-982d-efb4-de94ebc83f26) or [Excel Online](https://pages.store.office.com/addinsinstallpage.aspx?assetid=WA200007100&rs=en-US&correlationid=4132ecd1-425d-982d-efb4-de94ebc83f26&isWac=True). For more information, refer to [Microsoft Excel](/docs/cloud-integrations/semantic-layer/excel). - **New**: [Data health tile](/docs/collaborate/data-tile) is now generally available in dbt Explorer. Data health tiles provide a quick at-a-glance view of your data quality, highlighting potential issues in your data. You can embed these tiles in your dashboards to quickly identify and address data quality issues in your dbt project. diff --git a/website/docs/reference/resource-properties/constraints.md b/website/docs/reference/resource-properties/constraints.md index ff52a1fbcf4..948fe223d68 100644 --- a/website/docs/reference/resource-properties/constraints.md +++ b/website/docs/reference/resource-properties/constraints.md @@ -21,16 +21,61 @@ The structure of a constraint is: - `type` (required): one of `not_null`, `unique`, `primary_key`, `foreign_key`, `check`, `custom` - `expression`: Free text input to qualify the constraint. Required for certain constraint types, and optional for others. - `name` (optional): Human-friendly name for this constraint. Supported by some data platforms. -- `columns` (model-level only): List of column names to apply the constraint over +- `columns` (model-level only): List of column names to apply the constraint over. - + + +Foreign key constraints accept two additional inputs: +- `to`: A relation input, likely `ref()`, indicating the referenced table. +- `to_columns`: A list of column(s) in that table containing the corresponding primary or unique key. -When using `foreign_key`, you need to specify the referenced table's schema manually. Use `{{ target.schema }}` in the `expression` field to automatically pass the schema used by the target environment. Note that later versions of dbt will have more efficient ways of handling this. +This syntax for defining foreign keys uses `ref`, meaning it will capture dependencies and works across different environments. It's available in [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) and versions of dbt Core starting with v1.9. -For example: `expression: "{{ target.schema }}.customers(customer_id)"` + + +```yml +models: + - name: + + # required + config: + contract: {enforced: true} + + # model-level constraints + constraints: + - type: primary_key + columns: [first_column, second_column, ...] + - type: foreign_key # multi_column + columns: [first_column, second_column, ...] + to: "{{ ref('other_model_name') }}" + to_columns: [other_model_first_column, other_model_second_columns, ...] + - type: check + columns: [first_column, second_column, ...] + expression: "first_column != second_column" + name: human_friendly_name + - type: ... + + columns: + - name: first_column + data_type: string + + # column-level constraints + constraints: + - type: not_null + - type: unique + - type: foreign_key + to: "{{ ref('other_model_name') }}" + to_columns: other_model_column + - type: ... +``` + + + +In older versions of dbt Core, when defining a `foreign_key` constraint, you need to manually specify the referenced table in the `expression` field. You can use `{{ target }}` variables to make this expression environment-aware, but the dependency between this model and the referenced table is not captured. Starting in dbt Core v1.9, you can specify the referenced table using the `ref()` function. + ```yml @@ -39,44 +84,43 @@ models: # required config: - contract: - enforced: true + contract: {enforced: true} # model-level constraints constraints: - type: primary_key - columns: [FIRST_COLUMN, SECOND_COLUMN, ...] - - type: FOREIGN_KEY # multi_column - columns: [FIRST_COLUMN, SECOND_COLUMN, ...] - expression: "OTHER_MODEL_SCHEMA.OTHER_MODEL_NAME (OTHER_MODEL_FIRST_COLUMN, OTHER_MODEL_SECOND_COLUMN, ...)" + columns: [first_column, second_column, ...] + - type: foreign_key # multi_column + columns: [first_column, second_column, ...] + expression: "{{ target.schema }}.other_model_name (other_model_first_column, other_model_second_column, ...)" - type: check - columns: [FIRST_COLUMN, SECOND_COLUMN, ...] - expression: "FIRST_COLUMN != SECOND_COLUMN" - name: HUMAN_FRIENDLY_NAME + columns: [first_column, second_column, ...] + expression: "first_column != second_column" + name: human_friendly_name - type: ... columns: - - name: FIRST_COLUMN - data_type: DATA_TYPE + - name: first_column + data_type: string # column-level constraints constraints: - type: not_null - type: unique - type: foreign_key - expression: OTHER_MODEL_SCHEMA.OTHER_MODEL_NAME (OTHER_MODEL_COLUMN) + expression: "{{ target.schema }}.other_model_name (other_model_column)" - type: ... ``` - + ## Platform-specific support In transactional databases, it is possible to define "constraints" on the allowed values of certain columns, stricter than just the data type of those values. For example, Postgres supports and enforces all the constraints in the ANSI SQL standard (`not null`, `unique`, `primary key`, `foreign key`), plus a flexible row-level `check` constraint that evaluates to a boolean expression. -Most analytical data platforms support and enforce a `not null` constraint, but they either do not support or do not enforce the rest. It is sometimes still desirable to add an "informational" constraint, knowing it is _not_ enforced, for the purpose of integrating with legacy data catalog or entity-relation diagram tools ([dbt-core#3295](https://github.com/dbt-labs/dbt-core/issues/3295)). +Most analytical data platforms support and enforce a `not null` constraint, but they either do not support or do not enforce the rest. It is sometimes still desirable to add an "informational" constraint, knowing it is _not_ enforced, for the purpose of integrating with legacy data catalog or entity-relation diagram tools ([dbt-core#3295](https://github.com/dbt-labs/dbt-core/issues/3295)). Some data platforms can optionally use primary or foreign key constraints for query optimization if you specify an additional keyword. To that end, there are two optional fields you can specify on any filter: - `warn_unenforced: False` to skip warning on constraints that are supported, but not enforced, by this data platform. The constraint will be included in templated DDL. @@ -244,7 +288,7 @@ select Snowflake suppports four types of constraints: `unique`, `not null`, `primary key`, and `foreign key`. It is important to note that only the `not null` (and the `not null` property of `primary key`) are actually checked at present. -The rest of the constraints are purely metadata, not verified when inserting data. +The rest of the constraints are purely metadata, not verified when inserting data. Although Snowflake does not validate `unique`, `primary`, or `foreign_key` constraints, you may optionally instruct Snowflake to use them for query optimization by specifying [`rely`](https://docs.snowflake.com/en/user-guide/join-elimination) in the constraint `expression` field. Currently, Snowflake doesn't support the `check` syntax and dbt will skip the `check` config and raise a warning message if it is set on some models in the dbt project.