Skip to content

Commit

Permalink
Only allow manifest refresh if full-refresh flag is set
Browse files Browse the repository at this point in the history
  • Loading branch information
georgewoodhead committed Feb 15, 2024
1 parent 860d16e commit b450c7d
Show file tree
Hide file tree
Showing 3 changed files with 14 additions and 6 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,15 @@ snowplow-unified 0.2.1 (2024-02-XX)
## Summary
XXX

## 🚨 Breaking Changes 🚨
We have changed the behavior of the `allow_refresh` macro so now if `snowplow__allow_refresh` is set to `true` it will only refresh the manifest models if the `--full-refresh` flag is also set. If you require the old behavior where it would refresh the manifest models on an incremental run when `snowplow__allow_refresh` was set to `true`, please overwrite this macro. See the [Overriding Macros](https://docs.snowplow.io/docs/modeling-your-data/modeling-your-data-with-dbt/dbt-operation/macros-and-keys/#overriding-macros) guide for more details.

## Features
- Add new passthrough aggregations to the views, sessions, and users table, enabled using `snowplow__view/session/user_aggregations`
- Reorder and add some additional context fields to derived tables (non-breaking change)
- Add `snowplow__custom_sql` to allow adding custom sql to the `snowplow_unified_base_events_this_run` and `snowplow_unified_events_this_run` models
- Add macro to define cluster-by for tables to allow users to overwrite this if required
- Add check for `--full-refresh` flag before allowing refresh of manifest models when `snowplow__allow_refresh` is set to `true`.

## Fixes
- Fix a bug where if you ran the package in a period with no data, and had list all events enabled, the package would error rather than complete
Expand Down
2 changes: 1 addition & 1 deletion docs/markdown/snowplow_unified_macros_docs.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ The sql to extract the columns from the yauaa context, or these columns as nulls
This macro is used to determine if a full-refresh is allowed (depending on the environment), using the `snowplow__allow_refresh` variable.

#### Returns
`snowplow__allow_refresh` if environment is not `dev`, `none` otherwise.
`snowplow__allow_refresh` if environment is not `dev`, `none` otherwise. Returns `none` if the `--full-refresh` flag is not present.

{% endraw %}
{% enddocs %}
Expand Down
14 changes: 9 additions & 5 deletions macros/allow_refresh.sql
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,15 @@ You may obtain a copy of the Snowplow Personal and Academic License Version 1.0

{% macro default__allow_refresh() %}

{% set allow_refresh = snowplow_utils.get_value_by_target(
dev_value=none,
default_value=var('snowplow__allow_refresh'),
dev_target_name=var('snowplow__dev_target_name')
) %}
{% if flags.FULL_REFRESH == True %}
{% set allow_refresh = snowplow_utils.get_value_by_target(
dev_value=none,
default_value=var('snowplow__allow_refresh'),
dev_target_name=var('snowplow__dev_target_name')
) %}
{% else %}
{% set allow_refresh = none %}
{% endif %}

{{ return(allow_refresh) }}

Expand Down

0 comments on commit b450c7d

Please sign in to comment.