Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dbt config-blocks #64

Open
romeo-alexander opened this issue Jan 6, 2025 · 2 comments
Open

Add support for dbt config-blocks #64

romeo-alexander opened this issue Jan 6, 2025 · 2 comments

Comments

@romeo-alexander
Copy link

As far as I can tell, when it comes to model configs (eg. tags, pre/post hooks, materialization, enablement/disablement) in dbt-ibs, we can only specify them in either the dbt_project.yml project file, or a properties.yml file. Having dbt-ibs support the specification of these configs directly in the .ibis file, so that they are precompiled as a jinja dbt config block, would be very useful.

For example, it would be good to have the flexibility to specify configs in the following way:

@config(
    {
        "enabled": True if get_flags().target == "batch" else False,
    }
)
def model(table):
    return table

or

@config(
    {
        "materialized": "table" if get_flags().target == "batch" else "view",
    }
)
def model(table):
    return table

The idea is that the vast majority of the query logic applies to the multiple database contexts I may want to manager (specifically, batch and streaming), but to handle the parts that are different (eg. materialization, handling "now()"), being able to switch contexts directly in a config decorator would prevent having to maintain entirely separate dbt projects for each database context.

@binste
Copy link
Owner

binste commented Jan 10, 2025

Great idea, sounds very useful! Currently, I'm not spending much time on further feature development in dbt-ibis but I'm happy to review PRs if you or anyone else wants to give it a try.

  • config decorator sounds reasonable. It could simply attach the configurations to the function, similar to what depends_on does now
  • That could then be read out in
    def _get_ibis_expr_infos(
  • And then that information could be either written at the beginning of the generated SQL files somewhere at
    ibis_expr_info.sql_path.write_text(dbt_sql)
    or it could write separate .yml files (or just one?). If separate .yml files,
    _clean_up_unused_sql_files(
    should be extended to clean up unused .yml files.
  • A short mention in the docs would be nice

@romeo-alexander
Copy link
Author

Thanks for the suggestions. I generally find it easier to review dbt-SQL when the config blocks are in the sql file itself, so I'll try your first suggestion and put up a PR when done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants