-
Notifications
You must be signed in to change notification settings - Fork 128
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
incremental models failing in 1.9.0 and above versions #921
Comments
Thanks for reporting, investigating. |
Btw, I'm not sure this is relevant to the bug you are reporting, but we do not support job clusters for SQL execution. |
Can you share the stack trace found in the dbt.log for this error? Having trouble finding where we even attempt to call catalog on relation. |
@benc-db
I also looked at dbt.log but did not find more error specifications. |
Very strange... |
Is this still happening? I'm wondering if there is some upstream change, since I know they are working to introduce catalog (what we in Databricks call a metastore) as a type. Would it be possible to somehow print out all of the installed python libraries (particularly the versions of the dbt libraries)? Not sure how easy it is to do this in a Databricks Workflow. |
I think it's related to this: dbt-labs/dbt-adapters#286 If only we had a more helpful stack trace. |
@colin-rogers-dbt do you have any idea how to track this down? |
@SamrtCookie do you have any custom macros by any chance? |
@benc-db Yes I do. I have some macros example:
another example:
|
For now as a workaround I am using dbt-databricks==1.8.7 in my workflows. but yes the error is persisting. |
Could you try changing:
to
? |
Describe the bug
I have a running model with incremental materialization on Databricks:
I it is keep failing since I upgraded the dbt-databricks to version 1.9.1
Steps To Reproduce
I have created my source schemas as variables and target schemas for each model directory all in dbt-project.yml and catalogs in profiles.yml !
using compute with below configurations:
(I tried it firstly with lower run time versions for data bricks like 13 and after that as below I used 16 but no difference in the results
and using dbt-databricks 1.9.1 facing this error:
Expected behavior
As soon as I downgrade dbt-databricks to the 1.8.7 version everything will go well and error-free.
Screenshots and log output
System information
The output of
dbt --version
:The operating system you're using:
The output of
python --version
:3.10
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: