New features for hooks execution #367
nikita-sheremet-java-developer
started this conversation in
Ideas
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
There are several features to discuss. The goal of discussionis not only about needed features but also how to implement them.
Possibility to skip prehook execution when relation does not exist
Use case: sometime
detele+insert
are not sutable because a custom condition is needed. So it can be implemented aspre_hook=delete from {{this}} where id IN (select from ... join ... where ....)
. This aproach works, but for the first run prehook will fail because table does not exist.So adding feature to skip prehook execution when relation does not exist will sole this issue.
There is also question how to implement this? In all strategies or add new field to hook. (For now there is
sql
andinside_transaction
). For my opinioin add new field will be better.Possibility to run hook for suported materialization type
Use case: I have dbt, trino and iceberg. It is needed to clean up table snapshots, optimize files after operations (delete, update, inserts etc) and clean up orphal files. So it's better to add
post_hooks
to project level. But these hook will be applied to all models. if model/materialization type is nottable
will fail. So I have to place different models into different folders and copy and pastepost_hook
for all folders where table models are placed.So adding
materialization
field to hook will solve this issue - use can mark what hook will be applied on tables.Adding retry for hooks
Use case: I have dbt, trino and iceberg. And there is a but in Trino, that
optimize
operation may fail sometimes. A couple of retries "fixes" the issue. But for now I "use" airflow for retring. It is a good aproach, but during retry everything retried: prehooks, model, posthooks.So adding retry config for hook (regexp for error message, retry count) retrying will reduce total time for dbt model run.
Any opinions are welcome. Thanks in advance.
Beta Was this translation helpful? Give feedback.
All reactions