Running ALTER TABLE RECOVER PARTITIONS or MSCK REPAIR TABLE on a table that does not have partitions causes an error. So always running it is not an option.

I've created a quick fix to this issue by running ALTER TABLE RECOVER PARTITIONS if external.recover_partitions is true. This means that you can do this in your sources.yml:

external:
  recover_partitions: true

An even better alternative would be to use 'DESCRIBE TABLE' (or something similar) to determine if the table has partitions and run ALTER TABLE RECOVER PARTITIONS accordingly.
This would require changes to dbt-spark.

I can create a PR of my quick fix, if that is sufficient.

@jarno-r Definitely open to a PR for this one!

I like the idea of an explicit option for specifying that dbt + Databricks should recover/infer the partitions, when partitions is not itself defined.

I'm not strictly opposed to the cleverer approach, where dbt uses describe table to determine this on the user's behalf... but an explicit config feels in keeping with the approach on other databases that can infer partitions. As a general rule, I try to keep this package as a lightweight lens into each database's capabilities, without too many magic tricks behind the scenes.

I've created a PR. Link above.

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

@jtcohen6 This is still relevant for us. We've been running our own version of this package just to have this feature. It is cumbersome to maintain. Could the PR be merged?