Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: dbt-external-tables partitions #54

Conversation

betosaulo-cw
Copy link

@betosaulo-cw betosaulo-cw commented Dec 29, 2023

Description

This update makes changes to how we handle partitions in our project. Here's what's new:

  • We've added a partitions list in the ExternalConfig class. It's for keeping track of different sections of our data. By default, this list starts empty.
  • There's now a quick way to see partition details in ExternalConfig through partitions_map
  • We now consider partitions when we're organizing our data, making sure everything is in the right place.

Fixes #53

Checklist:

  • I have run make verify and fixed any linting or test errors
  • I have added appropriate unit tests or if applicable an integration test
  • OPTIONAL: I have run make integration against a Big Query instance

@ccharlesgb
Copy link
Collaborator

Looks like it will work! Can you add a test to check the dry run schema is as expected when partitions is used?

The test in here should be a good starting point but you will probably want to use assert_result_has_table like this to ensure the schema is correctly predicted:

# The test should pass if no InvalidColumnSpecification exception is raised
expected_table = Table(fields=[TableField(name="column1", type="STRING"), ..., ...])
assert_result_has_table(expected_table, result)

@ccharlesgb ccharlesgb self-requested a review January 5, 2024 09:23
Copy link
Collaborator

@ccharlesgb ccharlesgb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just needs a unit test on the source runner and then this is good 👍

@ccharlesgb
Copy link
Collaborator

Closing as this is stale. Would still accept a fresh PR 👍

@ccharlesgb ccharlesgb closed this May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] dbt-dry-run Ignoring Partitions from dbt-external-tables
2 participants