Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Snapshot time data type #243

Merged
merged 19 commits into from
Sep 4, 2024
+42 −0
Merged

Snapshot time data type #243

Show file tree
Hide file tree
Changes from 10 commits
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
dea83f4
Initial implementation of snapshot time data type comparisons
gshank Jun 20, 2024
7e8639d
Add call to warn_or_error for SnapshotTimestampWarning
gshank Jun 21, 2024
152a40a
Changie
gshank Jun 21, 2024
bd4b905
Merge branch 'main' into snapshot_time_data_type
gshank Jun 21, 2024
47d73c3
Merge branch 'main' into snapshot_time_data_type
gshank Jul 24, 2024
ec97052
Set dbt_updated_at_data_type to none
gshank Jul 25, 2024
e107a02
Merge branch 'main' into snapshot_time_data_type
gshank Jul 30, 2024
be0c620
Merge branch 'main' into snapshot_time_data_type
gshank Aug 21, 2024
c6c0321
Keep original signature for build_snapshot_staging_table
gshank Aug 21, 2024
1902060
Remove initial setting of dbt_updated_at_data_type
gshank Aug 27, 2024
3f1af9f
rename snapshot_get_time_data_type to get_snapshot_get_time_data_type
gshank Sep 3, 2024
e303ddd
Factor out time data type check
gshank Sep 3, 2024
777c500
remove unnecessary line
gshank Sep 3, 2024
a5e7a01
Fix variable name
gshank Sep 4, 2024
db51674
Merge branch 'main' into snapshot_time_data_type
gshank Sep 4, 2024
5dc54aa
Log snapshot_get_timestamp schema
gshank Sep 4, 2024
20441cc
Verify column schema gathering by logging
gshank Sep 4, 2024
823bd66
Check for upper case column name
gshank Sep 4, 2024
3887869
Remove logging, return none from macros
gshank Sep 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .changes/unreleased/Features-20240621-143024.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
kind: Features
body: Compare 'snapshot_get_time' and snapshot 'updated_at' data types
time: 2024-06-21T14:30:24.336219-04:00
custom:
Author: gshank
Issue: "242"
8 changes: 8 additions & 0 deletions dbt/include/global_project/macros/adapters/timestamps.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,14 @@
{{ current_timestamp() }}
{% endmacro %}

{% macro snapshot_get_time_data_type() %}
MichelleArk marked this conversation as resolved.
Show resolved Hide resolved
MichelleArk marked this conversation as resolved.
Show resolved Hide resolved
{% set snapshot_time = adapter.dispatch('snapshot_get_time', 'dbt')() %}
{% set time_data_type_sql = 'select ' ~ snapshot_time ~ ' as dbt_snapshot_time' %}
{% set snapshot_time_column_schema = get_column_schema_from_query(time_data_type_sql) %}
{% set time_data_type = snapshot_time_column_schema[0].dtype %}
{{ return(time_data_type) }}
{% endmacro %}

---------------------------------------------

/* {#
Expand Down
14 changes: 14 additions & 0 deletions dbt/include/global_project/macros/materializations/snapshots/helpers.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -179,3 +179,17 @@

{% do return(temp_relation) %}
{% endmacro %}


{% macro get_updated_at_column_data_type(snapshot_sql) %}
{% set snapshot_sql_column_schema = get_column_schema_from_query(snapshot_sql) %}
{% set dbt_updated_at_data_type = null %}
{% set ns = namespace() -%} {#-- handle for-loop scoping with a namespace --#}
{% set ns.dbt_updated_at_data_type = null -%}
{% for column in snapshot_sql_column_schema %}
{% if column.column == 'dbt_updated_at' %}
{% set ns.dbt_updated_at_data_type = column.dtype %}
{% endif %}
{% endfor %}
{{ return(ns.dbt_updated_at_data_type) }}
{% endmacro %}
8 changes: 8 additions & 0 deletions dbt/include/global_project/macros/materializations/snapshots/snapshot.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,12 +29,15 @@
{% if not target_relation_exists %}

{% set build_sql = build_snapshot_table(strategy, model['compiled_code']) %}
{% set dbt_updated_at_data_type = get_updated_at_column_data_type(build_sql) or none %}
mikealfare marked this conversation as resolved.
Show resolved Hide resolved
{% set final_sql = create_table_as(False, target_relation, build_sql) %}

{% else %}

{{ adapter.valid_snapshot_target(target_relation) }}

{% set snapshot_select_sql = snapshot_staging_table(strategy, sql, target_relation) %}
{% set dbt_updated_at_data_type = get_updated_at_column_data_type(snapshot_select_sql) or none %}
{% set staging_table = build_snapshot_staging_table(strategy, sql, target_relation) %}

-- this may no-op if the database does not require column expansion
Expand Down Expand Up @@ -71,6 +74,11 @@

{% endif %}

{% set get_time_data_type = snapshot_get_time_data_type() %}
{% if get_time_data_type is not none and dbt_updated_at_data_type is not none and get_time_data_type != dbt_updated_at_data_type %}
{{ exceptions.warn_snapshot_timestamp_data_types(get_time_data_type, dbt_updated_at_data_type) }}
{% endif %}

{% call statement('main') %}
{{ final_sql }}
{% endcall %}
Expand Down
Loading