Replace environment variable with a project flag to gate microbatch functionality #10799
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Thank you for your pull request! We could not find a changelog entry for this change. For details on how to document a change, see the contributing guide. |
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #10799 +/- ##
==========================================
- Coverage 89.08% 84.46% -4.63%
==========================================
Files 183 183
Lines 23580 23591 +11
==========================================
- Hits 21006 19925 -1081
- Misses 2574 3666 +1092
Flags with carried forward coverage won't be shown. Click here to find out more.
|
…yDefault` as it was prior to a9df50f
…with behavior flag changes This is needed to get the tests to pass. This is only necessary until the changes in dbt-adapters are merged. Said another way: we need to merge the changes to dbt-adapters first, and then revert these dependency changes before merging these changes.
QMalcolm
changed the title
QMalcolm
reviewed
Oct 1, 2024
Comment on lines
-1
to
+2
| git+https://github.com/dbt-labs/dbt-adapters.git@main | ||
| git+https://github.com/dbt-labs/dbt-adapters.git@main#subdirectory=dbt-tests-adapter | ||
| git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag | ||
| git+https://github.com/dbt-labs/dbt-adapters.git@microbatch-behavior-flag#subdirectory=dbt-tests-adapter |
There was a problem hiding this comment.
This is needed to get the tests to pass. This is only necessary until the changes in dbt-adapters are merged. Said another way: we need to merge the changes to dbt-adapters first, and then revert these dependency changes before merging this PR's changes.
4 tasks
MichelleArk
changed the title
…o `find_materialization_macro_candidate_by_name` This is necessary because in the following commit we're going to add a function to get whether or not a manifest should be run using the new microbatch functionality. Getting the MacroCandidate is necessary because we need to know the locality of the macro.
…ity should be used The new microbatch functionality is, unfortunately, potentially dangerous. That is it adds a new materalization strategy `microbatch` which an end user could have defined as a custom strategy previously. Additionally we added config keys to nodes, and as `config` is just a Dict[str, Any], it could contain anything, thus meaning people could already be using the configs we're adding for different purposes. Thus we need some intellegent gating. Specifically something that adheres to the following: cms = Custom Microbatch Strategy abms = Adapter Builtin Microbatch Strategy bf = Behavior flag umb = Use Microbatch Batching t/f/e = True/False/Error | cms | abms | bf | umb | | t | t | t | t | | f | t | t | t | | t | f | t | t | | f | f | t | e | | t | t | f | f | | f | t | f | t | | t | f | f | f | | f | f | f | e | (The above table assumes that there is a microbatch model present in the project) In order to achieve this we need to check that either the microbatch behavior flag is set to true OR microbatch materializaion being used is the _root_ microbatch materialization (i.e. not custom). The function we added in this commit, `use_microbatch_batches`, does just that.
…tom_microbatch_strategy`
QMalcolm
force-pushed
the
microbatch-project-flags
branch
from
9b3c122
to
543e024
Compare
In 0349968 I had done this for the function `find_materialization_macro_by_name`, but that wasn't the right function to do it to, and will be reverted shortly. `find_materialization_macro_by_name` is used for finding the general materialization macro, whereas `find_macro_by_name` is more general. For the work we're doing, we need to find the microbatch macro, which is not a materialization macro.
…_name` to `find_materialization_macro_candidate_by_name`" This reverts commit 0349968.
…tead of `root` Previously were were checking for a locality of `root`. However, a locality of `root` means it was provided by a `package`. We wnt to check for locality of `core` which basically means `builtin via dbt-core/adapters`. There is another locality `imported` which I beleive means it comes from another package.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolves #10798
Problem
We had gated the new microbatch feature behind an environment variable in the initial implementation of microbatch (#10594). However, for a better experience, we want people to be setting a project flag (AKA behavior flag).
Solution
Gate microbatch functionality behind a project flag (AKA behavior flag) instead of an environment variable.
Checklist