Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add required 'begin' config support for microbatch models #10756

Merged
merged 5 commits into from
Sep 24, 2024

Conversation

MichelleArk
Copy link
Contributor

@MichelleArk MichelleArk commented Sep 22, 2024

Resolves #10701

Problem

We need to know what the "beginning of time" is for a microbatch model, to know when to start for these two cases:

  • The first time this model is run
  • If you run in “full-refresh-mode”

This is distinct from the CLI parameter --event-time-start which can also be used when running in "incremental" mode.

Solution

  • continue to return early if --event-time-start is provided
  • For non-incremental runs (or if there is no checkpoint to lookback from), return the begin config! (raise exception if its not available, and further parse-time validation will be introduced as part of Microbatch Config Validation #10752
  • update schemas.getdbt.com v12 manifest
  • update tests to provide begin config.

Checklist

  • I have read the contributing guide and understand what's expected of me.
  • I have run this code in development, and it appears to resolve the stated issue.
  • This PR includes tests, or tests are not required or relevant for this PR.
  • This PR has no interface changes (e.g., macros, CLI, logs, JSON artifacts, config files, adapter interface, etc.) or this PR has already received feedback and approval from Product or DX.
  • This PR includes type annotations for new and modified functions.

@cla-bot cla-bot bot added the cla:yes label Sep 22, 2024
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

1 similar comment
Copy link
Contributor

Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide.

Copy link

codecov bot commented Sep 22, 2024

Codecov Report

Attention: Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.

Project coverage is 89.04%. Comparing base (db69473) to head (9e0f684).
Report is 5 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main   #10756      +/-   ##
==========================================
+ Coverage   88.97%   89.04%   +0.06%     
==========================================
  Files         181      181              
  Lines       22956    22984      +28     
==========================================
+ Hits        20424    20465      +41     
+ Misses       2532     2519      -13     
Flag Coverage Δ
integration 86.28% <88.88%> (+0.13%) ⬆️
unit 62.39% <88.88%> (-0.05%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
Unit Tests 62.39% <88.88%> (-0.05%) ⬇️
Integration Tests 86.28% <88.88%> (+0.13%) ⬆️

@MichelleArk
Copy link
Contributor Author

CI should pass once dbt-labs/schemas.getdbt.com#57 is merged

@MichelleArk MichelleArk marked this pull request as ready for review September 23, 2024 22:45
@MichelleArk MichelleArk requested a review from a team as a code owner September 23, 2024 22:45
Copy link
Contributor

@QMalcolm QMalcolm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woooo! This looks great 😄 I made one comment on updating a doc string, and we should probably fix it before merging, but also didn't want my review to be blocking as you'll be back online before I am.

@MichelleArk MichelleArk merged commit 730e40a into main Sep 24, 2024
66 checks passed
@MichelleArk MichelleArk deleted the microbatch-begin-config branch September 24, 2024 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
artifact_minor_upgrade To bypass the CI check by confirming that the change is not breaking cla:yes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Require defining a begin config for microbatch incremental models
2 participants