-
Notifications
You must be signed in to change notification settings - Fork 153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add an option to use INFORMATION_SCHEMA for partition info retrieval #866
base: main
Are you sure you want to change the base?
Add an option to use INFORMATION_SCHEMA for partition info retrieval #866
Conversation
19bd9b8
to
33167c1
Compare
6cc2b6c
to
9614b1c
Compare
The output contains the result from partitions information for your input table. | ||
The details of the retrieved columns can be found on https://cloud.google.com/bigquery/docs/managing-partitioned-tables | ||
It will leverage the INFORMATION_SCHEMA.PARTITIONS table. | ||
#} | ||
{%- macro get_partitions_metadata(table) -%} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was hoping to add an option to keep the previous approach but I can't since the prototype of adapter.get_partitions_metadata
is set in dbt-core and I don't think we want to leak the BQ specific option.
I think that, overall, it's acceptable.
9614b1c
to
4806a5d
Compare
4806a5d
to
d62d551
Compare
d62d551
to
717c275
Compare
Hi @github-christophe-oudar, we appreciate the effort you took to implement this optimization and we thank you for your patience. I've made sure this is properly filed in our queue of work but it may still take some time for us to give it a proper review. |
This PR has been marked as Stale because it has been open with no activity as of late. If you would like the PR to remain open, please comment on the PR or else it will be closed in 7 days. |
I don't think this should be closed, there's a long history of requests for review here and in the dbt bq slack channel. |
#867 got closed but I'm not sure it should be canceled. |
resolves #867
Problem
Current approach to retrieve partition information is expensive and slow.
Solution
Providing a way to use
INFORMATION_SCHEMA
metadata would speed up the partition info retrieval as well for a fraction of the cost (almost constant to 10 MB)Checklist