-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix: unrecognized _partitiontime for time ingestion partitioned models #74
fix: unrecognized _partitiontime for time ingestion partitioned models #74
Conversation
mode=BigQueryFieldMode.NULLABLE, | ||
) | ||
|
||
final_fields = [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How come we are removing the field from the predicted schema? If time ingestion partitioning is enabled then I can see how _PARTITIONTIME
gets added to the predicted schema but I don't see why you would remove the partition_by field from the predicted schema?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Assuming time_ingestion_partitioning
is set to true, I thought that there wouldn't be any queries that select the partition_by.field
since it essentially won't be a field in the real table. I put it back to the predicted schema.
result = self._replace_partition_with_time_ingestion_column(result) | ||
|
||
if not full_refresh: | ||
target_table = self._sql_runner.get_node_schema(node) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the incremental already exists does dbt behave differently? I don't think dbt can modify the partition keys for tables once they have been created without a full refresh
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I reverted this line back to the original.
a5f6b44
to
f78fcb7
Compare
f78fcb7
to
f8867cc
Compare
Seems like the latest python 3.12 version has broken something 🤔 Will look into this |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should be good to merge once you rebase from master 👍
Thanks 👍 |
This is release in v0.8.0. I think we still might be missing some edge cases here like if the table already exists but isn't partitioned and then someone adds |
Description
Update incremental runner to support models with time ingestion partitioning.
Fixes #73
Checklist:
make verify
and fixed any linting or test errorsmake integration
against a Big Query instance