Add microbatch strategy #924

QMalcolm · 2024-10-02T22:10:10Z

resolves #923

Problem

dbt-redshift needs a microbatch implementation that:

efficiently inserts new batches of data knowing that compiled_code will be filtered down by event_time
does not require a unique_key configuration on the model.

Solution

We did a custom implementation of delete+insert specifically for microbatch

Checklist

I have read the contributing guide and understand what's expected of me
I have run this code in development and it appears to resolve the stated issue
This PR includes tests, or tests are not required/relevant for this PR
This PR has no interface changes (e.g. macros, cli, logs, json artifacts, config files, adapter interface, etc) or this PR has already received feedback and approval from Product or DX

This work is basically in entirety a duplicate of the work done by MichelleArk in dbt-labs/dbt-snowflake#1179. I don't really expect this to work first try, but it might. I expect to need to do some edits, but who knows, maybe I'll get lucky.

… delete+insert

The `if` is unnecessary because predicates are guaranteed to exist, but the `if` was guarding against when there are no predicates.

mikealfare · 2024-11-06T22:09:39Z

Turning it off and on again for CI

MichelleArk · 2024-11-07T18:54:45Z

dbt/include/redshift/macros/materializations/incremental_merge.sql

+    {% endif %}
+
+    {#-- Add additional incremental_predicates to filter for batch --#}
+    {% do predicates.append(target ~ "." ~ model.config.event_time ~ " >= TIMESTAMP '" ~ model.config.__dbt_internal_microbatch_event_time_start ~ "'") %}


I think target is not strictly necessary here on this line or the one below since the delete from {{ target }} has no using clause so its unambiguous that model.config.event_time is a column on target

MichelleArk · 2024-11-07T18:55:07Z

dbt/include/redshift/macros/materializations/incremental_merge.sql

+        {% do predicates.append(pred) %}
+    {% endfor %}
+
+    {% if not model.config.get("__dbt_internal_microbatch_event_time_start") or not model.config.__dbt_internal_microbatch_event_time_end -%}


nit: not using model.config.get for the end time. let's be consistent across the two accesses

Good catch!

I was curious how this happened and did a little digging. In f7899e0 we removed the old separate if statements that added the predicates and added this new combined if statement. The old if statements came from the original snowflake implementation. I think did it that way originally because our initial implementation of batches in core the start time was not guaranteed. And so the get was probably to handle when the start time was None.

MichelleArk · 2024-11-07T19:12:37Z

Did some additional 🎩-ing locally in addition to the functional coverage we had for fun!

Initial build with 3 input rows --full-refresh

❯ dbt run --select +microbatch --full-refresh
19:05:08  Running with dbt=1.9.0-b3
19:05:08  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
19:05:09  Registered adapter: redshift=1.9.0-b1
19:05:09  [WARNING]: Time spines without YAML configuration are in the process of
deprecation. Please add YAML configuration for your 'metricflow_time_spine'
model. See documentation on MetricFlow time spines:
https://docs.getdbt.com/docs/build/metricflow-time-spine and behavior change
documentation:
https://docs.getdbt.com/reference/global-configs/behavior-changes.
19:05:09  Found 23 models, 6 seeds, 20 data tests, 6 sources, 2 exposures, 13 metrics, 799 macros, 6 semantic models, 5 unit tests
19:05:09  
19:05:09  Concurrency: 1 threads (target='dev')
19:05:09  
19:05:11  1 of 2 START sql view model michelle_ark.microbatch_input ...................... [RUN]
19:05:13  1 of 2 OK created sql view model michelle_ark.microbatch_input ................. [SUCCESS in 1.62s]
19:05:13  2 of 2 START sql microbatch model michelle_ark.microbatch ...................... [RUN]
19:05:13  1 of 12 START batch 2024-10-27 of michelle_ark.microbatch ...................... [RUN]
19:05:15  1 of 12 OK created batch 2024-10-27 of michelle_ark.microbatch ................. [SUCCESS in 1.70s]
19:05:15  2 of 12 START batch 2024-10-28 of michelle_ark.microbatch ...................... [RUN]
19:05:17  2 of 12 OK created batch 2024-10-28 of michelle_ark.microbatch ................. [SUCCESS in 1.82s]
19:05:17  3 of 12 START batch 2024-10-29 of michelle_ark.microbatch ...................... [RUN]
19:05:18  3 of 12 OK created batch 2024-10-29 of michelle_ark.microbatch ................. [SUCCESS in 1.93s]
19:05:18  4 of 12 START batch 2024-10-30 of michelle_ark.microbatch ...................... [RUN]
19:05:20  4 of 12 OK created batch 2024-10-30 of michelle_ark.microbatch ................. [SUCCESS in 1.84s]
19:05:20  5 of 12 START batch 2024-10-31 of michelle_ark.microbatch ...................... [RUN]
19:05:22  5 of 12 OK created batch 2024-10-31 of michelle_ark.microbatch ................. [SUCCESS in 1.84s]
19:05:22  6 of 12 START batch 2024-11-01 of michelle_ark.microbatch ...................... [RUN]
19:05:24  6 of 12 OK created batch 2024-11-01 of michelle_ark.microbatch ................. [SUCCESS in 1.50s]
19:05:24  7 of 12 START batch 2024-11-02 of michelle_ark.microbatch ...................... [RUN]
19:05:25  7 of 12 OK created batch 2024-11-02 of michelle_ark.microbatch ................. [SUCCESS in 1.67s]
19:05:25  8 of 12 START batch 2024-11-03 of michelle_ark.microbatch ...................... [RUN]
19:05:27  8 of 12 OK created batch 2024-11-03 of michelle_ark.microbatch ................. [SUCCESS in 1.58s]
19:05:27  9 of 12 START batch 2024-11-04 of michelle_ark.microbatch ...................... [RUN]
19:05:29  9 of 12 OK created batch 2024-11-04 of michelle_ark.microbatch ................. [SUCCESS in 1.62s]
19:05:29  10 of 12 START batch 2024-11-05 of michelle_ark.microbatch ..................... [RUN]
19:05:30  10 of 12 OK created batch 2024-11-05 of michelle_ark.microbatch ................ [SUCCESS in 1.60s]
19:05:30  11 of 12 START batch 2024-11-06 of michelle_ark.microbatch ..................... [RUN]
19:05:32  11 of 12 OK created batch 2024-11-06 of michelle_ark.microbatch ................ [SUCCESS in 1.63s]
19:05:32  12 of 12 START batch 2024-11-07 of michelle_ark.microbatch ..................... [RUN]
19:05:33  12 of 12 OK created batch 2024-11-07 of michelle_ark.microbatch ................ [SUCCESS in 1.64s]
19:05:33  2 of 2 OK created sql microbatch model michelle_ark.microbatch ................. [SUCCESS in 20.43s]
19:05:34  
19:05:34  Finished running 1 incremental model, 1 view model in 0 hours 0 minutes and 24.60 seconds (24.60s).
19:05:34  
19:05:34  Completed successfully
19:05:34  
19:05:34  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

❯ dbt show --inline "select * from {{ ref('microbatch') }}"
19:05:43  Running with dbt=1.9.0-b3
19:05:43  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
19:05:43  Registered adapter: redshift=1.9.0-b1
19:05:44  Found 23 models, 6 seeds, 20 data tests, 1 sql operation, 6 sources, 2 exposures, 13 metrics, 799 macros, 6 semantic models, 5 unit tests
19:05:44  
19:05:44  Concurrency: 1 threads (target='dev')
19:05:44  
19:05:46  Previewing inline node:
| x | event_time |
| - | ---------- |
| 3 | 2024-10-30 |
| 1 | 2024-10-28 |
| 2 | 2024-10-29 |

Adding a couple input rows that are out of bounds + incremental run

❯ dbt run --select +microbatch
19:05:55  Running with dbt=1.9.0-b3
User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
19:05:55  Registered adapter: redshift=1.9.0-b1
19:05:56  Found 23 models, 6 seeds, 20 data tests, 6 sources, 2 exposures, 13 metrics, 799 macros, 6 semantic models, 5 unit tests
19:05:56  
19:05:56  Concurrency: 1 threads (target='dev')
19:05:56  
19:06:00  1 of 2 OK created sql view model michelle_ark.microbatch_input ................. [SUCCESS in 1.88s]
19:06:00  2 of 2 START sql microbatch model michelle_ark.microbatch ...................... [RUN]
19:06:00  1 of 2 START batch 2024-11-06 of michelle_ark.microbatch ....................... [RUN]
19:06:03  1 of 2 OK created batch 2024-11-06 of michelle_ark.microbatch .................. [SUCCESS in 2.94s]
19:06:03  2 of 2 START batch 2024-11-07 of michelle_ark.microbatch ....................... [RUN]
19:06:06  2 of 2 OK created batch 2024-11-07 of michelle_ark.microbatch .................. [SUCCESS in 2.54s]
19:06:06  2 of 2 OK created sql microbatch model michelle_ark.microbatch ................. [SUCCESS in 5.51s]
19:06:06  Finished running 1 incremental model, 1 view model in 0 hours 0 minutes and 10.26 seconds (10.26s).
19:06:06  
19:06:06  Completed successfully
19:06:06  
19:06:06  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

❯ dbt show --inline "select * from {{ ref('microbatch') }}"
19:06:10  Running with dbt=1.9.0-b3
User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
19:06:10  Registered adapter: redshift=1.9.0-b1
19:06:11  Found 23 models, 6 seeds, 20 data tests, 1 sql operation, 6 sources, 2 exposures, 13 metrics, 799 macros, 6 semantic models, 5 unit tests
19:06:11  
19:06:11  Concurrency: 1 threads (target='dev')
19:06:11  
19:06:14  Previewing inline node:
| x | event_time |
| - | ---------- |
| 2 | 2024-10-29 |
| 3 | 2024-10-30 |
| 1 | 2024-10-28 |

Adding a row of new + current data, incremental run

❯ dbt run --select +microbatch
19:06:30  Running with dbt=1.9.0-b3
19:06:30  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
19:06:30  Registered adapter: redshift=1.9.0-b1
19:06:31  [WARNING]: Time spines without YAML configuration are in the process of
deprecation. Please add YAML configuration for your 'metricflow_time_spine'
model. See documentation on MetricFlow time spines:
https://docs.getdbt.com/docs/build/metricflow-time-spine and behavior change
documentation:
https://docs.getdbt.com/reference/global-configs/behavior-changes.
19:06:31  Found 23 models, 6 seeds, 20 data tests, 6 sources, 2 exposures, 13 metrics, 799 macros, 6 semantic models, 5 unit tests
19:06:31  
19:06:31  Concurrency: 1 threads (target='dev')
19:06:31  
19:06:33  1 of 2 START sql view model michelle_ark.microbatch_input ...................... [RUN]
19:06:35  1 of 2 OK created sql view model michelle_ark.microbatch_input ................. [SUCCESS in 1.78s]
19:06:35  2 of 2 START sql microbatch model michelle_ark.microbatch ...................... [RUN]
19:06:35  1 of 2 START batch 2024-11-06 of michelle_ark.microbatch ....................... [RUN]
19:06:37  1 of 2 OK created batch 2024-11-06 of michelle_ark.microbatch .................. [SUCCESS in 2.18s]
19:06:37  2 of 2 START batch 2024-11-07 of michelle_ark.microbatch ....................... [RUN]
19:06:39  2 of 2 OK created batch 2024-11-07 of michelle_ark.microbatch .................. [SUCCESS in 1.79s]
19:06:39  2 of 2 OK created sql microbatch model michelle_ark.microbatch ................. [SUCCESS in 3.99s]
19:06:40  
19:06:40  Finished running 1 incremental model, 1 view model in 0 hours 0 minutes and 8.48 seconds (8.48s).
19:06:40  
19:06:40  Completed successfully
19:06:40  
19:06:40  Done. PASS=2 WARN=0 ERROR=0 SKIP=0 TOTAL=2

❯ dbt show --inline "select * from {{ ref('microbatch') }}"
19:06:42  Running with dbt=1.9.0-b3
19:06:42  [WARNING]: Deprecated functionality

User config should be moved from the 'config' key in profiles.yml to the 'flags' key in dbt_project.yml.
19:06:43  Registered adapter: redshift=1.9.0-b1
19:06:43  Found 23 models, 6 seeds, 20 data tests, 1 sql operation, 6 sources, 2 exposures, 13 metrics, 799 macros, 6 semantic models, 5 unit tests
19:06:43  
19:06:43  Concurrency: 1 threads (target='dev')
19:06:43  
19:06:45  Previewing inline node:
| x | event_time |
| - | ---------- |
| 1 | 2024-10-28 |
| 2 | 2024-10-29 |
| 3 | 2024-10-30 |
| 6 | 2024-11-07 |

MichelleArk

Manual testing done after I applied the couple nits locally. Looks great!

…in microbatch materialization The `target.` portion of `target.<column_name>` is unnecessary for the predicates in the microbatch materialization macro because the delete statement already ensures the "targeting` of `target` in the delete statement via the clause `delete from {{ target }}`. Said another way, there is no use of the word `using` in the delete clause, thus it is unambiguous what is being deleted from.

Add microbatch strategy

3ca6459

This work is basically in entirety a duplicate of the work done by MichelleArk in dbt-labs/dbt-snowflake#1179. I don't really expect this to work first try, but it might. I expect to need to do some edits, but who knows, maybe I'll get lucky.

cla-bot bot added the cla:yes label Oct 2, 2024

QMalcolm added the Skip Changelog Skips GHA to check for changelog file label Oct 2, 2024

dbt-labs deleted a comment from github-actions bot Oct 2, 2024

QMalcolm added 3 commits October 2, 2024 17:11

Add changie doc

2f557ab

Add comment to microbatch macro to explain why we are re-implementing…

db3d8e3

… delete+insert

Add begin to microbatch config in test_incremental_microbatch.py

070d5cb

QMalcolm removed the Skip Changelog Skips GHA to check for changelog file label Oct 3, 2024

QMalcolm and others added 5 commits October 3, 2024 15:09

Cleanup predicates in microbatch materialization

84f2ec8

Merge branch 'main' into qmalcolm--microbatch-strategy

a71d2e1

Fix predicate/incremental predicate logic in microbatch macro

f7899e0

Remove unnecessary if in microbatch macro

57b25e0

The `if` is unnecessary because predicates are guaranteed to exist, but the `if` was guarding against when there are no predicates.

Merge branch 'main' into qmalcolm--microbatch-strategy

b6bf7fc

QMalcolm marked this pull request as ready for review November 6, 2024 19:10

QMalcolm requested a review from a team as a code owner November 6, 2024 19:10

colin-rogers-dbt closed this Nov 6, 2024

colin-rogers-dbt reopened this Nov 6, 2024

Merge branch 'main' into qmalcolm--microbatch-strategy

f5a9996

mikealfare approved these changes Nov 6, 2024

View reviewed changes

mikealfare closed this Nov 6, 2024

mikealfare reopened this Nov 6, 2024

Merge branch 'main' into qmalcolm--microbatch-strategy

f1d21c4

MichelleArk reviewed Nov 7, 2024

View reviewed changes

MichelleArk approved these changes Nov 7, 2024

View reviewed changes

QMalcolm added 2 commits November 7, 2024 15:24

Get batch start and end time in the same way

60e91b8

QMalcolm enabled auto-merge (squash) November 7, 2024 22:33

QMalcolm merged commit fccbe2d into main Nov 7, 2024
37 checks passed

QMalcolm deleted the qmalcolm--microbatch-strategy branch November 7, 2024 22:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add microbatch strategy #924

Add microbatch strategy #924

QMalcolm commented Oct 2, 2024 •

edited

Loading

mikealfare commented Nov 6, 2024

MichelleArk Nov 7, 2024

MichelleArk Nov 7, 2024

QMalcolm Nov 7, 2024 •

edited

Loading

MichelleArk commented Nov 7, 2024 •

edited

Loading

MichelleArk left a comment

Add microbatch strategy #924

Add microbatch strategy #924

Conversation

QMalcolm commented Oct 2, 2024 • edited Loading

Problem

Solution

Checklist

mikealfare commented Nov 6, 2024

MichelleArk Nov 7, 2024

Choose a reason for hiding this comment

MichelleArk Nov 7, 2024

Choose a reason for hiding this comment

QMalcolm Nov 7, 2024 • edited Loading

Choose a reason for hiding this comment

MichelleArk commented Nov 7, 2024 • edited Loading

Initial build with 3 input rows --full-refresh

Adding a couple input rows that are out of bounds + incremental run

Adding a row of new + current data, incremental run

MichelleArk left a comment

Choose a reason for hiding this comment

QMalcolm commented Oct 2, 2024 •

edited

Loading

QMalcolm Nov 7, 2024 •

edited

Loading

MichelleArk commented Nov 7, 2024 •

edited

Loading