Investigate options for dbt failure notifications

[jkarpen]

Goal for this issue is to explore different options for having dbt cloud failure notifications go to a space where multiple team members (including client agencies ideally) can triage and assign the fix.

• Slack integration (failure creates a Slack message visible to the whole DSE team)
• Teams notification (failure creates a Teams message, may require having a shared email group, ideally this would also be visible to agency teams) (Not currently viable)
• Email notification (the failure sends to a shared email group with both DSE and client staff CC'd) <- lower priority, Slack/Teams are preferred options.

[summer-mothwood]

@britt-allen I ran into this tutorial for using dbt webhooks for sending notifications to Teams when troubleshooting the Slack integration -- do we already use Zapier (or any other web automation tool that can work with webhooks)? https://docs.getdbt.com/guides/zapier-ms-teams?step=1

[summer-mothwood]

Looking more into dbt Cloud's webhook capabilities (and reviewing @ram-kishore-odi 's write up for Andrew on a somewhat similar issue) -- it looks like Microsoft Teams has some built-in functionality for subscribing to webhooks, too, so we might not even need the Zapier intermediate step!

[britt-allen]

We could send an email to a channel if we were not on the 365 Gov't plan...

[britt-allen]

This option could work, but I cannot install the required app because I had to make a request which I imagine goes to an Admin at GovOps. Lots of our ability to explore is blocked by ODI not having its own Microsoft license yet.

[britt-allen]

I have checked off Teams as not viable for now.

[summer-mothwood]

I'm looking into setting up Slack notification in dbt cloud. The first step is to set up Slack as an Account Integration in dbt Cloud. When I try to click through here

I get this error

Looking into it further (using these instructions https://docs.getdbt.com/docs/deploy/job-notifications#slack-notifications), I notice the docs say

You must be an account admin to configure Slack notifications in dbt Cloud.

I tried changing my license to an IT license, but still got the same error. @britt-allen do you know if one of us is an account admin on the dbt cloud account that has elevated privileges for that kind of thing? or should I put in a request ticket with dbt?

[britt-allen]

Let me give it a try!

[britt-allen]

I get the same error as well. I think this is because how IT has setup our dbt account, I'd go to our IT team first before opening a ticket with dbt @summer-mothwood

[summer-mothwood]

Thanks for looking into it @britt-allen !! I'll reach out to IT -- we'll need a system admin on the Slack side to complete the next steps anyway, so I'll ask them about that while I'm at it too

[summer-mothwood]

Put in a ticket to IT to integrate Slack and dbt for us.

[summer-mothwood]

I met with Kevin in IT today, and he was unable to configure Slack notifications / add dbt as an integrated app into Slack either. He's going to follow up with Chad and will notify me (in Slack, hah!) with updates. I mentioned again that our preference would be to enable notifications in Teams in order to make it easy for our clients who don't use Slack. Keeping the status here as blocked.

[summer-mothwood]

Chad ran into the same issue, so Kevin filed a ticket with dbt. Response from their team here, Kevin is working on implementing this morning:

Thank you for your patience. So it appears that there is an additional step that needs to be taken for Slack integration given your account is hosted on a Single Tenant instance. Would you be able to kindly do the following:

Create a Slack application here: https://api.slack.com/apps
Once the above is completed, kindly provide the client ID and client secret. Please ensure to send the Client ID in plaintext and provide the Client secret using a self destructing service such as https://1ty.me/ for security purposes.

We're also looking into adding the additional requirement to our documentation here: https://docs.getdbt.com/docs/deploy/job-notifications#slack-notifications.

[britt-allen]

Listservs grouping together external clients and ODI team members are possible. We'll just need to submit a request to IT at the start of the engagement. So I think this can be a good interim solution. See this thread here for a bit more context: https://cadotgov.slack.com/archives/C03SSD8FN8G/p1732299331165939

[summer-mothwood]

Swinging back around to the Slack side of things -- we're seeing new Oauth errors when trying to get the Slack app connected to our Caldata dbt account. Kevin is continuing to work with dbt on that front, and I've looped Taylor from dbt into that support ticket too for visibility.

Exciting that listservs are an option for us though! @jkarpen I say we add 'set up a listserv' as an issue for the coming weeks, along with 'set up new fivetran account', etc. We can create the listserv now and just add us DSE folks to it for now, then supply IT with a list of HFA folks to add to it when we're much closer to kick off (which we already have an issue for https://github.com/cagov/caldata-doe-mdsa-hfa-marketing/issues/29)

[jkarpen]

@summer-mothwood I created this issue for the Listserv: https://github.com/cagov/caldata-doe-mdsa-hfa-marketing/issues/27

@ian-r-rose are you already creating issues for setting up Fivetran and other accounts for HFA? If not I can start those.

[ian-r-rose]

I have not yet created those issues, please do! Ian Rose (he/him) Principal Data Engineer, Data Services | CA Office of Data and Innovation (916) 931-8386 | ***@***.*** Follow us: web <https://innovation.ca.gov/> | blog <https://innovation.ca.gov/blog/> | twitter <https://twitter.com/californiaodi>

…

On Thu, Dec 5, 2024 at 3:19 PM Joshua Karpen ***@***.***> wrote: @summer-mothwood <https://github.com/summer-mothwood> I created this issue for the Listserv: cagov/caldata-doe-mdsa-hfa-marketing#27 <cagov/caldata-doe-mdsa-hfa-marketing#27> @ian-r-rose <https://github.com/ian-r-rose> are you already creating issues for setting up Fivetran and other accounts for HFA? If not I can start those. — Reply to this email directly, view it on GitHub <#415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABLWQN5E5ALVHBK35RXRK3L2EDNPPAVCNFSM6AAAAABQTXMCRCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMRRG4YDGNBYG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jkarpen]

Closing this issue as we have investigated and found a path forward. Might create a new issue for tracking the follow-on task of getting the dbt/slack integration set up but that is more on the IT side.