Joining the modern data stack with the modern ML stack

[sungchun12]

Contact Details

[email protected] (@jacopotagliabue on Slack)

This issue is being created to promote an approved Discussion.

• I am promoting this issue from an approved Discussion and will link it below.

Link to the Discussion this Issue is promoting.

This started from my conversation with @sungchun12 wrt to a sequel to his great article: https://docs.getdbt.com/blog/maching-learning-dbt-baton-pass. In particular, we discussed bringing together dbt and Metaflow to show how dbt can be leveraged by teams running ML pipelines on Metaflow, to retrieve datasets and features prepared with SQL.

More generally, this is part of our OS evangelization for ML "at reasonable scale" (https://towardsdatascience.com/mlops-without-much-ops-d17f502f76e8), including DataOps (https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat).

If you're promoting this from an 'I want to write about...' Discussion, you can copy over the answers to the following questions. Otherwise, please answer these questions below

1. Combining dbt (and dbt cloud) with Metaflow (and SageMaker, and Comet) in a principled way. In particular, a major drawback of our pipeline (https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat) is its complexity - we want to show a minimalist version of how dataOps and MLops play together: simple enough to be easy to try, complex enough to be realistic;
2. dbt is a powerful feature preparation language - if you subscribe to the end-2-end data scientist, preparing features is a crucial part of the job, and we show how that can be done inside the same Python tool you use for the ML part (Metaflow)
3. Building a ML pipeline, going from raw data in Snowflake, to run dbt transformation, to query a feature table, training a model and then deploying it. A small but complete life cycle of a modern data scientist job.

Link to the initial outline we will develop through this Issue.

Some notes written for our TDS series are here (https://docs.google.com/document/d/1eebJZQ2yy096nfWIfoYnvBDnmmeW-itwHkLItzQgt-g/edit). A working repo has already been shared with @sungchun12.

[sungchun12]

@jacopotagliabue I migrated your issue to a discussion to follow the devhub blog workflow

[gwenwindflower]

thanks, sung!

[gwenwindflower]

@Jacopo if you take a look at this template for a I want to write about... Discussion -- which is where we start knowledge development and can follow the initial instructions there to develop an outline of the piece, we'll work from there. you can hopefully just edit the initial issue body above with the Discussion questions and answers. thanks!

[sungchun12]

@jacopotagliabue

[jacopotagliabue]

@gwenwindflower - my answers below: let me know if you need anything else .

What is the main problem you are solving? What is your solution?

DataOps are a crucial part of Data-centric AI, but not the only one: as transformation is rarely a goal in itself, MLOps are still needed to create value from data. We show a principled, end to end ML pipeline combining best in class tools for Data (Snowflake, dbt) and MLOps (Metaflow, Reclist, Comet, etc.)

Why should the reader care about this problem? Why is your solution the right one?

There are many tutorials explaining how ML pipelines or data transformations work: rarely however, you see all the pieces of the puzzle in a coherent picture, working at scale with real-world (non-toy) data. If you are setting up a ML practice right now, our solution gives you production quality tooling to go from millions of data points to a working endpoint, without allocating any devOps resource. Unlike many tutorials, this solution is tried and tested at unicorn scale, and comes with free data to test it out.

Can you list the steps of your solution for the reader here?

Leverage Metaflow to define data and ML work as a DAG in pure Python. While that is in place:

1. run dbt (local or cloud!) to prepare feature tables
2. get data from Snowflake
3. seamlessly spin up GPUs to train a neural network in the cloud
4. run tests
5. deploy a model in AWS

Are there other existing solutions that solve the problem, and if so, how is this solution better or different? If so please share any links here.

There is a mildly popular repo (https://github.com/jacopotagliabue/you-dont-need-a-bigger-boat) we published last year at RecSys, which contains a first version of these ideas. However, we often get the feedback that the learning curve is too steep: too many tools, too many moving pieces. The proposed stack is still a e-2-e version, but simpler version of the same idea. Check also the many refs in the article for further context.

LINK (ORIGINAL DRAWING CAN BE FOUND IN A LINK ON TOP OF THE DOC): https://docs.google.com/document/d/1eebJZQ2yy096nfWIfoYnvBDnmmeW-itwHkLItzQgt-g/edit

[sungchun12]

@gwenwindflower @jacopotagliabue

We are aiming to release this blog early May, is this reasonable to achieve?

[jasnonaz]

Hi @jacopotagliabue - thanks so much for putting together this awesome post! Very excited to get this live - a couple thoughts on how we can tweak this to be a super impactful fit for the dev blog.

• I was extraordinarily compelled by the section about unifying BI and data science workflows at Coveo! I'm wondering if we can make that more prominent - would love the focus here to be on the way you unified these workflows and lessons for folks reading it - happy to include the parts about the stack that was used, just think it will resonate even stronger when framed in the real world reason you adopted it.
• I found myself a little lost in the sections describing MLops and the data science workflows parts of this - wondering if we can give some extra context for the analytics engineers reading this who might not be as familiar with data science workflow.
• I'm curious if we're worried that the post-modern data stack title might confuse folks given Drew's talk with the same name

Once we've got these three things hammered out we can move this over to an issue and get ready to publish!

[jacopotagliabue]

Hi @jasnonaz , you can reach me on dbt slack if it's easier to chat. Quickly here:

• sure, do you want to comment on the doc the parts you feel are superfluous and the ones you think instead should be expanded?
• of course, happy to add more context and explain the ML part a bit more!
• oh, I had no idea about that, thx for bringing it to my attention - you suggest to just leave it at "Joining the modern data stack with a modern ML stack?" or want to re-phrase the title and keep the subtitle?

Ping me on Slack and we can refine things there!

[johnblust]

Hey @jacopotagliabue, thanks for bearing with us as we've worked through this. We think this is good content, but unfortunately, this isn't a good fit for the dbt Developer Blog in its current state.

The dbt Developer Blog is meant to be a platform for analytics engineer practitioners to share their personal experiences and solutions to challenges with other practitioners. We don't always meet that ideal, but we're trying to get as close to it as we can. There are areas of this content that really resonate with that standard, but there are large portions of the content that sway too far towards traditional content marketing (ie, explorations of industry trends and tool stacks).

If you're open to it, the team and I can go through section-by-section to approve or disapprove sections of the content as being acceptable for this platform next week. But if you'd want to publish this content as-is, the dbt Developer Blog isn't the best platform for this piece. Let me know how you'd like to move forward. Thanks!