Liquid clustered table creation is failing when running together with model in insert_overwrite incremental strategy under one run session

[kevinkevin1]

Describe the bug

1. Databricks has dynamic partition overwrite config (here) which allow us to implement insert_overwrite incremental strategy.

2. DBT set spark.sql.sources.partitionOverwriteMode to DYNAMIC during the incremental step of table with insert_overwrite strategy (dbt-databricks source code).

3. This can create conflict for liquid clustered table down the line within the same DBT running session. Liquid clustered table couldn’t be created if dynamic partition overwrite mode is enabled. It will trigger the following error DELTA_CLUSTERING_WITH_DYNAMIC_PARTITION_OVERWRITE.

Steps To Reproduce

1. Create a liquid clustered model with upstream table updated with insert_overwrite strategy & incremental materialization.
2. Run dbt command with single thread to force it in one single session.

dbt run -s +liquid_clustered_model --threads 1

Expected behavior

DBT should always run set spark.sql.sources.partitionOverwriteMode = STATIC as a pre-hook for liquid clustered model to ensure it will always avoid this kind of failure.

This can be fixed easily by set the same syntax above as liquid cluster model's pre-hook. However, DBT should be able to handle this automatically under the hood.

Screenshots and log output

none

System information

The output of dbt --version:

Core:
  - installed: 1.8.4
  - latest:    1.9.2 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.8.3 - Update available!
  - spark:      1.8.0 - Update available!

  At least one plugin is out of date or incompatible with dbt-core.
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Operating system: MacOS Sonoma 14.6.1

Python version: Python 3.11.11

Additional context

none

[benc-db]

Does this behavior still exist in 1.9.x?