Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate options for dbt failure notifications #415

Closed
2 of 3 tasks
jkarpen opened this issue Oct 25, 2024 · 17 comments
Closed
2 of 3 tasks

Investigate options for dbt failure notifications #415

jkarpen opened this issue Oct 25, 2024 · 17 comments
Assignees

Comments

@jkarpen
Copy link
Collaborator

jkarpen commented Oct 25, 2024

Goal for this issue is to explore different options for having dbt cloud failure notifications go to a space where multiple team members (including client agencies ideally) can triage and assign the fix.

  • Slack integration (failure creates a Slack message visible to the whole DSE team)
  • Teams notification (failure creates a Teams message, may require having a shared email group, ideally this would also be visible to agency teams) (Not currently viable)
  • Email notification (the failure sends to a shared email group with both DSE and client staff CC'd) <- lower priority, Slack/Teams are preferred options.
@summer-mothwood
Copy link
Contributor

@britt-allen I ran into this tutorial for using dbt webhooks for sending notifications to Teams when troubleshooting the Slack integration -- do we already use Zapier (or any other web automation tool that can work with webhooks)? https://docs.getdbt.com/guides/zapier-ms-teams?step=1

@summer-mothwood
Copy link
Contributor

Looking more into dbt Cloud's webhook capabilities (and reviewing @ram-kishore-odi 's write up for Andrew on a somewhat similar issue) -- it looks like Microsoft Teams has some built-in functionality for subscribing to webhooks, too, so we might not even need the Zapier intermediate step!

@britt-allen
Copy link
Contributor

We could send an email to a channel if we were not on the 365 Gov't plan... Image

@britt-allen
Copy link
Contributor

This option could work, but I cannot install the required app because I had to make a request which I imagine goes to an Admin at GovOps. Lots of our ability to explore is blocked by ODI not having its own Microsoft license yet.

@britt-allen
Copy link
Contributor

I have checked off Teams as not viable for now.

@summer-mothwood
Copy link
Contributor

I'm looking into setting up Slack notification in dbt cloud. The first step is to set up Slack as an Account Integration in dbt Cloud. When I try to click through here
Image

I get this error
Image

Looking into it further (using these instructions https://docs.getdbt.com/docs/deploy/job-notifications#slack-notifications), I notice the docs say

You must be an account admin to configure Slack notifications in dbt Cloud.

I tried changing my license to an IT license, but still got the same error. @britt-allen do you know if one of us is an account admin on the dbt cloud account that has elevated privileges for that kind of thing? or should I put in a request ticket with dbt?

@britt-allen
Copy link
Contributor

Let me give it a try!

@britt-allen
Copy link
Contributor

I get the same error as well. I think this is because how IT has setup our dbt account, I'd go to our IT team first before opening a ticket with dbt @summer-mothwood

@summer-mothwood
Copy link
Contributor

Thanks for looking into it @britt-allen !! I'll reach out to IT -- we'll need a system admin on the Slack side to complete the next steps anyway, so I'll ask them about that while I'm at it too

@summer-mothwood
Copy link
Contributor

Put in a ticket to IT to integrate Slack and dbt for us.

@summer-mothwood
Copy link
Contributor

I met with Kevin in IT today, and he was unable to configure Slack notifications / add dbt as an integrated app into Slack either. He's going to follow up with Chad and will notify me (in Slack, hah!) with updates. I mentioned again that our preference would be to enable notifications in Teams in order to make it easy for our clients who don't use Slack. Keeping the status here as blocked.

@summer-mothwood
Copy link
Contributor

Chad ran into the same issue, so Kevin filed a ticket with dbt. Response from their team here, Kevin is working on implementing this morning:

Thank you for your patience. So it appears that there is an additional step that needs to be taken for Slack integration given your account is hosted on a Single Tenant instance. Would you be able to kindly do the following:

Create a Slack application here: https://api.slack.com/apps
Once the above is completed, kindly provide the client ID and client secret. Please ensure to send the Client ID in plaintext and provide the Client secret using a self destructing service such as https://1ty.me/ for security purposes.

We're also looking into adding the additional requirement to our documentation here: https://docs.getdbt.com/docs/deploy/job-notifications#slack-notifications.

@britt-allen
Copy link
Contributor

Listservs grouping together external clients and ODI team members are possible. We'll just need to submit a request to IT at the start of the engagement. So I think this can be a good interim solution. See this thread here for a bit more context: https://cadotgov.slack.com/archives/C03SSD8FN8G/p1732299331165939

@summer-mothwood
Copy link
Contributor

summer-mothwood commented Dec 3, 2024

Swinging back around to the Slack side of things -- we're seeing new Oauth errors when trying to get the Slack app connected to our Caldata dbt account. Kevin is continuing to work with dbt on that front, and I've looped Taylor from dbt into that support ticket too for visibility.

Exciting that listservs are an option for us though! @jkarpen I say we add 'set up a listserv' as an issue for the coming weeks, along with 'set up new fivetran account', etc. We can create the listserv now and just add us DSE folks to it for now, then supply IT with a list of HFA folks to add to it when we're much closer to kick off (which we already have an issue for https://github.com/cagov/caldata-doe-mdsa-hfa-marketing/issues/29)

@jkarpen
Copy link
Collaborator Author

jkarpen commented Dec 5, 2024

@summer-mothwood I created this issue for the Listserv: https://github.com/cagov/caldata-doe-mdsa-hfa-marketing/issues/27

@ian-r-rose are you already creating issues for setting up Fivetran and other accounts for HFA? If not I can start those.

@ian-r-rose
Copy link
Member

ian-r-rose commented Dec 5, 2024 via email

@jkarpen
Copy link
Collaborator Author

jkarpen commented Dec 10, 2024

Closing this issue as we have investigated and found a path forward. Might create a new issue for tracking the follow-on task of getting the dbt/slack integration set up but that is more on the IT side.

@jkarpen jkarpen closed this as completed Dec 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants