Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] copy_partitions with static partitions list produces empty statement #1349

Open
2 tasks done
alexandru-pisarenco-dh opened this issue Sep 18, 2024 · 1 comment · May be fixed by #1351
Open
2 tasks done

[Bug] copy_partitions with static partitions list produces empty statement #1349

alexandru-pisarenco-dh opened this issue Sep 18, 2024 · 1 comment · May be fixed by #1351
Labels
bug Something isn't working community This PR is from a community member incremental

Comments

@alexandru-pisarenco-dh
Copy link

Is this a new bug in dbt-bigquery?

  • I believe this is a new bug in dbt-bigquery
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

When an incremental load with copy_partitions=True, and partitions=[... list of partitions ...] is used, the macro bq_static_insert_overwrite_sql does not produce any SQL statements, which leads the BigQuery client to trigger an error.

On top of that, because this code:

  -- 2. clean up the temp table
  drop table if exists {{ tmp_relation }};

is only executed when copy_partitions=False, the temp table is not deleted.

Expected Behavior

A non-empty statement should be produced by the macro and sent to dbt. I figure that drop table if exists {{tmp_relation}}; should just be taken out of any conditional blocks and just always be there.

Steps To Reproduce

  1. Create a new model with the following config:
{{
  config(
    materialized="incremental",
    partition_by={
      "field": "partition_date",
      "data_type": "date",
      "granularity": "day",
      "copy_partitions": true,
    },
    incremental_strategy="insert_overwrite",
    partitions=["2024-08-13", "2024-08-14"]
  )
}}
  1. Run the model

Relevant log output

14:48:05  Completed with 1 error and 0 warnings:
14:48:05  
14:48:05    Database Error in model the_model (models/the_model.sql)
  Syntax error: Unexpected end of statement at [1:331]
  compiled Code at /Users/alex/the_project/target/run/the_project/models/the_model.sql

Environment

- OS: MacOS
- Python: 3.10.14
- dbt-core: 1.8.6
- dbt-bigquery: 1.8.2

Additional Context

No response

@alexandru-pisarenco-dh alexandru-pisarenco-dh added bug Something isn't working triage labels Sep 18, 2024
@Kayrnt Kayrnt linked a pull request Sep 21, 2024 that will close this issue
4 tasks
@github-christophe-oudar
Copy link
Contributor

github-christophe-oudar commented Sep 21, 2024

It wasn't supported but I wrote #1351 to do so.
As a workaround, you might just use the updated macros and use them as a macro override to check in your own dbt project.

@colin-rogers-dbt colin-rogers-dbt added the community This PR is from a community member label Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working community This PR is from a community member incremental
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants