Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New snapshot_meta_column_names config for dbt snapshots #6211

Merged
merged 43 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
f6a5780
Getting started on docs for `snapshot_meta_column_names` config
dbeatty10 Oct 2, 2024
008d818
Merge branch 'add/new-snapshot-spec-main' into dbeatty10-patch-1
mirnawong1 Oct 2, 2024
c4ead5f
The metadata columns for snapshots can be customized via the `snapsho…
dbeatty10 Oct 2, 2024
53bd2ff
Add the new `snapshot_meta_column_names` config to the release notes
dbeatty10 Oct 2, 2024
a5208f8
Skeleton reference page for `snapshot_meta_column_names` config for s…
dbeatty10 Oct 2, 2024
fcd5ecd
Links to `snapshot_meta_column_names` page across project file, prope…
dbeatty10 Oct 2, 2024
3d3e09d
Add new page to `sidebars.js`
dbeatty10 Oct 2, 2024
98a66b4
Rename `snapshot_meta_column_names` to `snapshot_meta_column_names.md`
dbeatty10 Oct 2, 2024
97aa3a2
Add hyperlinks for property YAML file `schema.yml` example
dbeatty10 Oct 2, 2024
aceaaf9
Uniform newlines with the code examples for project and property YAML…
dbeatty10 Oct 2, 2024
5090922
Link to the reference page for the `snapshot_meta_column_names` config
dbeatty10 Oct 2, 2024
e21dad5
Remove the `<Version>` tag so that it is always visible
dbeatty10 Oct 2, 2024
74991e6
Merge branch 'add/new-snapshot-spec-main' into dbeatty10-patch-1
dbeatty10 Oct 2, 2024
0fe2acd
Rough draft for `snapshot_meta_column_names` config
dbeatty10 Oct 2, 2024
31b4cfc
Remove the reference to the `dbt_valid_to_current` config until it is…
dbeatty10 Oct 2, 2024
911d83a
Remove extraneous content
dbeatty10 Oct 2, 2024
6781603
Add a filename for the example
dbeatty10 Oct 2, 2024
dc9746f
Align with other snapshot configs that lead with Jinja and project fi…
dbeatty10 Oct 2, 2024
faf60df
Add an example of the table output
dbeatty10 Oct 2, 2024
fa38b7b
Update release-notes.md
dbeatty10 Oct 2, 2024
2bb58f5
Add a property file / `schema.yml` example to the top
dbeatty10 Oct 2, 2024
28d84e5
Fix SQL file docs for Jinja config
dbeatty10 Oct 2, 2024
5e67484
Available in v1.9 or with versionless dbt Cloud
dbeatty10 Oct 2, 2024
4cc1efc
Link to SCD type 2 wikipedia page
dbeatty10 Oct 2, 2024
f28e48c
Fix hyperlink
dbeatty10 Oct 2, 2024
28e51d3
Link to page for building dbt snapshots
dbeatty10 Oct 2, 2024
f9dc63e
Add `datatype` and `id` for this docs page
dbeatty10 Oct 2, 2024
a55ab99
Add `default_value` for this page
dbeatty10 Oct 2, 2024
fb73bc3
Merge branch 'add/new-snapshot-spec-main' into dbeatty10-patch-1
dbeatty10 Oct 2, 2024
3e36fc3
Link to the metadata fields specifically
dbeatty10 Oct 2, 2024
cc8175e
Link to dbt Cloud Versionless
dbeatty10 Oct 2, 2024
2ac9a9f
Starting in 1.9
dbeatty10 Oct 2, 2024
f9ba325
Starting in v1.9
dbeatty10 Oct 2, 2024
3409771
Update snapshot_meta_column_names.md
dbeatty10 Oct 2, 2024
36c1547
Separate release note
dbeatty10 Oct 2, 2024
e47e381
Version entire page
dbeatty10 Oct 2, 2024
0b2c49b
Improve wording and fix misspellings
dbeatty10 Oct 2, 2024
464ce51
Convert from a note to a warning
dbeatty10 Oct 2, 2024
5467e1f
Update wording in release notes
dbeatty10 Oct 2, 2024
54e9f06
Merge branch 'add/new-snapshot-spec-main' into dbeatty10-patch-1
dbeatty10 Oct 2, 2024
af80431
Merge branch 'current' into dbeatty10-patch-1
dbeatty10 Oct 3, 2024
a9f25e3
Remove extraneous newline
dbeatty10 Oct 3, 2024
54153b4
Merge branch 'current' into dbeatty10-patch-1
dbeatty10 Oct 3, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions website/docs/docs/build/snapshots.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,6 +512,8 @@ Basically – keep your query as simple as possible! Some reasonable exceptions

Snapshot <Term id="table">tables</Term> will be created as a clone of your source dataset, plus some additional meta-fields*.

In v1.9 or with [dbt Cloud Versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless), these column names can be customized to your team or organizational conventions via the [`snapshot_meta_column_names`](/reference/resource-configs/snapshot_meta_column_names) config.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

| Field | Meaning | Usage |
| -------------- | ------- | ----- |
| dbt_valid_from | The timestamp when this snapshot row was first inserted | This column can be used to order the different "versions" of a record. |
Expand Down
2 changes: 1 addition & 1 deletion website/docs/docs/dbt-versions/release-notes.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Release notes are grouped by month for both multi-tenant and virtual private clo
## October 2024


- **New**: In dbt Cloud Versionless, [Snapshots](/docs/build/snapshots) have been updated to use YAML configuration files instead of SQL snapshot blocks. This new feature simplifies snapshot management and improves performance, and will soon be released in dbt Core 1.9.
- **New**: In dbt Cloud Versionless, [Snapshots](/docs/build/snapshots) have been updated to use YAML configuration files instead of SQL snapshot blocks. This new feature simplifies snapshot management and improves performance, and will soon be released in dbt Core 1.9. This includes a new `snapshot_meta_column_names` config for customizing the snapshot metadata columns.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved
- Who does this affect? New user on Versionless can define snapshots using the new YAML specification. Users upgrading to Versionless who use snapshots can keep their existing configuration or can choose to migrate their snapshot definitions to YAML.
- Users on dbt 1.8 and earlier: No action is needed; existing snapshots will continue to work as before. However, we recommend upgrading to Versionless to take advantage of the new snapshot features.
- **Enhancement**: In dbt Cloud Versionless, snapshots defined in SQL files can now use `config` defined in `schema.yml` YAML files. This update resolves the previous limitation that required snapshot properties to be defined exclusively in `dbt_project.yml` and/or a `config()` block within the SQL file. This enhancement will be included in the upcoming dbt Core v1.9 release.
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
---
resource_types: [snapshots]
description: "Snapshot meta column names"
datatype: "{<dictionary>}"
default_value: {"dbt_valid_from": "dbt_valid_from", "dbt_valid_to": "dbt_valid_to", "dbt_scd_id": "dbt_scd_id", "dbt_updated_at": "dbt_updated_at"}
id: "snapshot_meta_column_names"
---

Available in v1.9 or with [versionless](/docs/dbt-versions/upgrade-dbt-version-in-cloud#versionless) dbt Cloud.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

<File name='snapshots/schema.yml'>

```yaml
snapshots:
- name: <snapshot_name>
config:
snapshot_meta_column_names:
dbt_valid_from: <string>
dbt_valid_to: <string>
dbt_scd_id: <string>
dbt_updated_at: <string>

```

</File>

<File name='snapshots/<filename>.sql'>

```jinja2
{{
config(
snapshot_meta_column_names={
"dbt_valid_from": "<string>",
"dbt_valid_to": "<string>",
"dbt_scd_id": "<string>",
"dbt_updated_at": "<string>",
}
)
}}

```

</File>

<File name='dbt_project.yml'>

```yml
snapshots:
[<resource-path>](/reference/resource-configs/resource-path):
+snapshot_meta_column_names:
dbt_valid_from: <string>
dbt_valid_to: <string>
dbt_scd_id: <string>
dbt_updated_at: <string>

```

</File>

## Description

In order to align with an organization's naming conventions, the `snapshot_meta_column_names` config can be used to customize the names of the [metadata columns](/docs/build/snapshots#snapshot-meta-fields) within each snapshot.

## Default

By default, dbt snapshots use the following column names to track change history using [Type 2 slowly changing dimension](https://en.wikipedia.org/wiki/Slowly_changing_dimension#Type_2:_add_new_row) records:

| Field | Meaning | Notes |
| -------------- | ------- | ----- |
| `dbt_valid_from` | The timestamp when this snapshot row was first inserted and became valid. | The value is affected by the [`strategy`](/reference/resource-configs/strategy). |
| `dbt_valid_to` | The timestamp when this row is no longer valid. | |
| `dbt_scd_id` | A unique key generated for each snapshot row. | This is used internally by dbt. |
| `dbt_updated_at` | The `updated_at` timestamp of the source record when this snapshot row was inserted. | This is used internally by dbt. |

However, these column names can be customized using the `snapshot_meta_column_names` config.

:::note
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

To avoid any uninentional data modification, dbt will **not** automatically apply any column renames. So if a user applyies `snapshot_meta_column_names` config for a snapshot without updating the pre-existing table, they will get an error. Our recommendation is to either only use these settings for net-new snapshots, or to arrange an update of pre-existing tables prior to committing a column name change.
dbeatty10 marked this conversation as resolved.
Show resolved Hide resolved

:::

## Example

<File name='snapshots/schema.yml'>

```yaml
snapshots:
- name: orders_snapshot
relation: ref("orders")
config:
unique_key: id
strategy: check
check_cols: all
snapshot_meta_column_names:
dbt_valid_from: start_date
dbt_valid_to: end_date
dbt_scd_id: scd_id
dbt_updated_at: modified_date
```

</File>

The resulting snapshot table contains the configured meta column names:

| id | scd_id | modified_date | start_date | end_date |
| -- | -------------------- | -------------------- | -------------------- | -------------------- |
| 1 | 60a1f1dbdf899a4dd... | 2024-10-02 ... | 2024-10-02 ... | 2024-10-02 ... |
| 2 | b1885d098f8bcff51... | 2024-10-02 ... | 2024-10-02 ... | |
4 changes: 3 additions & 1 deletion website/docs/reference/snapshot-configs.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,7 @@ snapshots:
[+](/reference/resource-configs/plus-prefix)[strategy](/reference/resource-configs/strategy): timestamp | check
[+](/reference/resource-configs/plus-prefix)[updated_at](/reference/resource-configs/updated_at): <column_name>
[+](/reference/resource-configs/plus-prefix)[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all
[+](/reference/resource-configs/plus-prefix)[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}

```

Expand Down Expand Up @@ -100,6 +101,7 @@ snapshots:
[strategy](/reference/resource-configs/strategy): timestamp | check
[updated_at](/reference/resource-configs/updated_at): <column_name>
[check_cols](/reference/resource-configs/check_cols): [<column_name>] | all
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names): {<dictionary>}

```
</File>
Expand All @@ -113,7 +115,6 @@ snapshots:
<VersionBlock lastVersion="1.8">

```jinja

{{ config(
[target_schema](/reference/resource-configs/target_schema)="<string>",
[target_database](/reference/resource-configs/target_database)="<string>",
Expand All @@ -139,6 +140,7 @@ snapshots:
[strategy](/reference/resource-configs/strategy)="timestamp" | "check",
[updated_at](/reference/resource-configs/updated_at)="<column_name>",
[check_cols](/reference/resource-configs/check_cols)=["<column_name>"] | "all"
[snapshot_meta_column_names](/reference/resource-configs/snapshot_meta_column_names)={<dictionary>}
) }}

```
Expand Down
1 change: 1 addition & 0 deletions website/sidebars.js
Original file line number Diff line number Diff line change
Expand Up @@ -974,6 +974,7 @@ const sidebarSettings = {
"reference/resource-configs/unique_key",
"reference/resource-configs/updated_at",
"reference/resource-configs/invalidate_hard_deletes",
"reference/resource-configs/snapshot_meta_column_names",
],
},
{
Expand Down
Loading