Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Iceberg support for view materialization #341

Open
3 tasks done
dataders opened this issue Nov 1, 2024 · 0 comments
Open
3 tasks done

[Feature] Iceberg support for view materialization #341

dataders opened this issue Nov 1, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@dataders
Copy link

dataders commented Nov 1, 2024

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-adapters functionality, rather than a Big Idea better suited to a discussion

Describe the feature

with dbt-labs/dbt-snowflake#321, dbt-snowflake now supports Iceberg table format with table_format="iceberg" for table, incremental, dynamic, and snapshot materializations. Other adapters already support Iceberg table format, and will hopefully soon standardize on the same config.

However, there is now a spec for Iceberg views. While not yet supported in any of the adapters that we support (except for Vanilla Spark), I expect that we'll see them eventually.

Perhaps this means that table_format might also soon be an option we'd want to extend to the view materialization. Maybe it's view_format, but I think it makes more sense to be consistent in config naming so that table_format may be set for a group of tables & views that a user desires to write to an Iceberg catalog.

{{
  config(
    materialized="view",
    table_format="iceberg"
)}}

Describe alternatives you've considered

No response

Who will this benefit?

from @Fokko:

With Iceberg having view support now, I think dbt and Iceberg are going to be a great fit and is going to open up the analytics landscape where you can do the heavy lifting with one engine and do the final analytics with another
The version of views currently is still bound to the engine that makes them (it is just SQL that's being stored), but we're transitioning into a intermediate-representation like Substrait, but that's going to need a bit more time/work

Are you interested in contributing this feature?

No response

Anything else?

No response

@dataders dataders added enhancement New feature or request triage labels Nov 1, 2024
@amychen1776 amychen1776 removed the triage label Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants