Deprecating Custom Normalization (dbt) in Airbyte OSS

[rileybrook]

👋 Hello Airbyte Community!

Summary

Today, we are announcing that Custom Normalization (dbt) will be deprecated from future versions of Airbyte OSS.

Support will end on March 31st 🙏

Reasoning behind the change

• This year, we are doubling down on Airbyte’s reliability and making it even better. Part of this is ensuring all parts of the product work as intended. Where that is not the case, we will invest in improving that feature or invest that time elsewhere. Investing that time into functionality which is even more critical to ensuring you have a fantastic user experience.

• We are seeing that a tiny minority of Airbyte OSS users utilize our Custom Normalization feature to trigger dbt jobs. Custom Normalization is an area that is taking time away from other great features we could implement for the community. Deprecating this feature exemplifies how we are choosing to invest less in one area to invest more heavily in other platform areas.

• We have seen the use cases for Airbyte OSS expand into mature production deployments on massive scales. Instead of trying to do too many things at once, we are focused on executing the fundamentals of Extract & Load very, very well. We are focused on providing an incredibly reliable product experience for teams of all sizes using Airbyte OSS in 2024 and beyond.

What this means for the future

• Your use case is still supported! Airbyte integrates with leading orchestration tools. Our tutorials and quickstarts for Airflow, Dagster, and Prefect show you how to set this up quickly and easily. Additionally, we are surveying the community on which tools they use to trigger dbt jobs. We’d love your survey feedback in slack here! We will use this survey to identify which orchestration tools are critical to have tutorials & documentation for.

• We will circle back to this area. In the future, we plan to eventually support “post-sync” webhooks that trigger on sync completion. We have not committed to a timeline for this, nor will it be in the immediate future (2024). That said, there will be more to come down the road.

[cgardens]

This deprecation was completed with release 0.57.0! Will leave this discussion open for a couple days in case there are any questions.

[leo-schick]

I think it is a pity that post-hook webhooks are not planned in the intermediate future 😭

[cgardens]

Thanks for your note, @leo-schick ! Post hooks is definitely a feature that we will introduce, but I don't know the timeline yet. As noted, in the meanwhile you can still use several free, open-source orchestration platforms and / or the airbyte api to get the desired effect.

[dramirez-bluon]

So sad :'c

I was really looking forward this functionality, hopefully the webhook can be scheduled earlier.

[JCWahoo]

This is disappointing - The whole reason we use custom dbt normalization is that Airbyte is failing to dedupe and has poor performance on its normalization

[killthekitten]

It's a bit confusing that you can still add transformations to a V2 connection when setting it up and the docs are not clear on the deprecation.

[Jean-Roc]

Hi @rileybrook , your project's readme still list transformation with SQL or DBT in Getting Started, your documentation points to this discussion where you list data orchestrators as replacements but as a new comer, it is not clear where I should look in airbyte for the T in ELT.

[lc-bi-data]

This is a huge bummer. We've got dozens of connections that will need to be retooled due to this change.

[djpirra]

I also believe normalization is a quite relevant feature for the initial staging specially flattening JSON substructures.