Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding ADO service principal #6647

Open
wants to merge 9 commits into
base: current
Choose a base branch
from
Open

Adding ADO service principal #6647

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
c37d7f4
Adding ADO service principal
matthewshaver Dec 12, 2024
c409a58
Update website/docs/docs/cloud/git/setup-azure.md
matthewshaver Dec 12, 2024
6416673
Apply suggestions from code review
matthewshaver Dec 12, 2024
567b64f
Adding release note
matthewshaver Dec 12, 2024
d315b7e
Merge branch 'service-principal' of https://github.com/dbt-labs/docs.…
matthewshaver Dec 12, 2024
03d71fc
Fixing syntax
matthewshaver Dec 12, 2024
78421a9
Merge branch 'current' into service-principal
matthewshaver Dec 12, 2024
910be92
Finishing migration
matthewshaver Dec 12, 2024
478b204
Merge branch 'service-principal' of https://github.com/dbt-labs/docs.…
matthewshaver Dec 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
115 changes: 92 additions & 23 deletions website/docs/docs/cloud/git/setup-azure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,10 +17,10 @@
4. [Connect Azure DevOps to your new app](#connect-azure-devops-to-your-new-app).
5. [Add your Entra ID app to dbt Cloud](#add-your-azure-ad-app-to-dbt-cloud).

Once the Microsoft Entra ID app is added to dbt Cloud, an account admin must also [connect a service user](/docs/cloud/git/setup-azure#connect-a-service-user) via OAuth, which will be used to power headless actions in dbt Cloud such as deployment runs and CI.
Once the Microsoft Entra ID app is added to dbt Cloud, an account admin must also connect a [service principal](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals?tabs=browser), which will be used to power headless actions in dbt Cloud such as deployment runs and CI.


Once the Microsoft Entra ID app is added to dbt Cloud and the service user is connected, then dbt Cloud developers can personally authenticate in dbt Cloud from Azure DevOps. For more on this, see [Authenticate with Azure DevOps](/docs/cloud/git/authenticate-azure).
Once the Microsoft Entra ID app is added to dbt Cloud and the service principal is connected, then dbt Cloud developers can personally authenticate in dbt Cloud from Azure DevOps. For more on this, see [Authenticate with Azure DevOps](/docs/cloud/git/authenticate-azure).

The following personas are required to complete the steps on this page:
- Microsoft Entra ID admin
Expand Down Expand Up @@ -62,22 +62,20 @@

## Add another redirect URI

A Microsoft Entra ID admin needs to add another redirect URI to your Entra ID application. This redirect URI will be used to authenticate the service user for headless actions in deployment environments.
A Microsoft Entra ID admin needs to add another redirect URI to your Entra ID application. This redirect URI will be used to authenticate the service principal for headless actions in deployment environments.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With service principal I don't think we need to add redirect uri to the entra app anymore. But if they are switching from service user to service principal with the same application credentials, they can just keep the redirect uri


1. Navigate to your Microsoft Entra ID application.

2. Select the link next to **Redirect URIs**
3. Click **Add URI** and add the URI, replacing `YOUR_ACCESS_URL` with the [appropriate Access URL](/docs/cloud/about-cloud/access-regions-ip-addresses) for your region and plan:
`https://YOUR_ACCESS_URL/complete/azure_active_directory_service_user`
`https://YOUR_ACCESS_URL/complete/azure_active_directory_service_principal`
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably don't need this part either. But you could mention perhaps that if they are switching from service user to service principal they don't need to remove the redirect uris. Probably preferred to not remove them if they want to switch back to service user.

4. Click **Save**.

<Lightbox src="/img/docs/dbt-cloud/connecting-azure-devops/redirect-uri.gif" title="Adding the Service User redirect URI"/>

## Create a client secret

A Microsoft Entra ID admin needs to complete the following steps:

1. Navigate to your Microsoft Entra ID application.
1. Navigate to **Microsoft Entra ID**, click **App registrations**, and click on your app.
2. Select **Certificates and Secrets** from the left navigation panel.
3. Select **Client secrets** and click **New client secret**
4. Give the secret a description and select the expiration time. Click **Add**.
Expand All @@ -89,7 +87,7 @@
- Azure Service Administrator
- Azure Co-administrator

If your Azure DevOps account is connected to Entra ID, then you can proceed to [Connect a service user](#connect-a-service-user). However, if you're just getting set up, connect Azure DevOps to the Microsoft Entra ID app you just created:
If your Azure DevOps account is connected to Entra ID, then you can proceed to [Connect a service principal](#connect-a-service-principal). However, if you're just getting set up, connect Azure DevOps to the Microsoft Entra ID app you just created:

1. From your Azure DevOps account, select **Organization settings** in the bottom left.
2. Navigate to Microsoft Entra ID.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part is incorrect for service principal (unless these are for service user). For service principal, you go to org settings -> navigate to users -> add users -> search and add your application as a service principal in the search bar -> add projects for Add to projects and also for Azure DevOps Groups select Project Administrators (this is important for being able to create webhooks)

Expand All @@ -99,29 +97,79 @@

<Lightbox src="/img/docs/dbt-cloud/connecting-azure-devops/connect AD to Azure DevOps.gif" title="Connecting Azure DevOps and Microsoft Entra ID"/>

## Add your Microsoft Entra ID app to dbt Cloud
## Configure the Entra ID connection

Check warning on line 100 in website/docs/docs/cloud/git/setup-azure.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/docs/cloud/git/setup-azure.md#L100 

[custom.SentenceCaseHeaders] 'Configure the Entra ID connection' should use sentence-style capitalization. Try '' instead.
Raw output 
{"message": "[custom.SentenceCaseHeaders] 'Configure the Entra ID connection' should use sentence-style capitalization. Try '' instead.", "location": {"path": "website/docs/docs/cloud/git/setup-azure.md", "range": {"start": {"line": 100, "column": 4}}}, "severity": "WARNING"}

A dbt Cloud account admin needs to perform the following steps.
There are two connection methods currently available for dbt Cloud and Azure DevOps:
- **Service principal** (recommended): Create an application connection via client ID and secret for unattended authentication.
- **Service user** (legacy): Create a user that will authenticate the connection with username and password. This configuration should be avoided.

Once you connect your Microsoft Entra ID app and Azure DevOps, you need to provide dbt Cloud information about the app:
<Tabs>

<TabItem value="Service principal">

1. Navigate to your account settings in dbt Cloud.
2. Select **Integrations**.
3. Scroll to the Azure DevOps section.
4. Complete the form:
- **Azure DevOps Organization:** Must match the name of your Azure DevOps organization exactly. Do not include the `dev.azure.com/` prefix in this field. ✅ Use `my-devops-org` ❌ Avoid `dev.azure.com/my-devops-org`
- **Application (client) ID:** Found in the Microsoft Entra ID app.
- **Client Secrets:** Copy the **Value** field in the Microsoft Entra ID app client secrets and paste it in the **Client Secret** field in dbt Cloud. Entra ID admins are responsible for the Entra ID app secret expiration and dbt Admins should note the expiration date for rotation.
- **Directory(tenant) ID:** Found in the Microsoft Entra ID app.
<Lightbox src="/img/docs/dbt-cloud/connecting-azure-devops/AzureDevopsAppdbtCloud.gif" title="Adding a Microsoft Entra ID app to dbt Cloud"/>
## Create a service principal

Your Microsoft Entra ID app should now be added to your dbt Cloud Account. People on your team who want to develop in the dbt Cloud IDE or dbt Cloud CLI can now personally [authorize Azure DevOps from their profiles](/docs/cloud/git/authenticate-azure).
The application's service principal represents the Entra ID application object. Whereas a service user represents a real user in Azure with an Entra ID (and an applicable license), the service principal is a secure identity used by an application to access Azure resources unattended. The service principal authenticates with a client ID and secret rather than a username and password (or any other form of user auth). Service principals are the [Microsoft recommended method](https://learn.microsoft.com/en-us/entra/architecture/secure-service-accounts#types-of-microsoft-entra-service-accounts) for authenticating apps.

### Add a role to the Service Principal

Check warning on line 114 in website/docs/docs/cloud/git/setup-azure.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/docs/cloud/git/setup-azure.md#L114 

[custom.SentenceCaseHeaders] 'Add a role to the Service Principal' should use sentence-style capitalization. Try '' instead.
Raw output 
{"message": "[custom.SentenceCaseHeaders] 'Add a role to the Service Principal' should use sentence-style capitalization. Try '' instead.", "location": {"path": "website/docs/docs/cloud/git/setup-azure.md", "range": {"start": {"line": 114, "column": 5}}}, "severity": "WARNING"}

You can create a new role or assign an existing one to the service principal app. It must have the following permissions:

- **Project Reader**
- **ViewSubscriptions**
- **EditSubscriptions**
- **DeleteSubscriptions** *
- **PullRequestContribute**
- **GenericContribute**

matthewshaver marked this conversation as resolved.
Show resolved Hide resolved
In your Azure account:

1. Navigate to **Subscriptions** and click on the appropriate subscription name for the application environment.
2. From the left-side menu of the subscription window, click **Access control (IAM)**.
3. From the top menu, click **Add** and select **Add role assignment** from the dropdown.

<Lightbox src="/img/docs/cloud-integrations/azure-subscription.png" width="60%" title="The 'Access control (IAM)' window in the 'Subscriptions' section of Azure."/>

4. In the **Role** tab, select a role with appropriate permissions to assign the service principal.
5. Click the **Members** tab. You must set **Assign access to** to **User, group, or service principal**.

Check warning on line 134 in website/docs/docs/cloud/git/setup-azure.md

View workflow job for this annotation

GitHub Actions / vale

[vale] website/docs/docs/cloud/git/setup-azure.md#L134 

[custom.Repitition] 'to' is repeated!
Raw output 
{"message": "[custom.Repitition] 'to' is repeated!", "location": {"path": "website/docs/docs/cloud/git/setup-azure.md", "range": {"start": {"line": 134, "column": 1}}}, "severity": "WARNING"}
6. Click **Select members** and search for your app name in the window. Once it appears, click your app, which will appear in the **Selected members** section. Click **Select** at the bottom to save your selection.

<Lightbox src="/img/docs/cloud-integrations/assign-app-to-members.png" width="80%" title="The dbt Cloud ADO app in the members section."/>

5. Confirm the details are correct and click **Review + assign**.

<Lightbox src="/img/docs/cloud-integrations/review-and-assign.png" width="80%" title="Review screen with the app service principal and permissions."/>

Navigate back to the **App registrations** screen and click the app. On the left menu, click **Roles and administrators** and you will see the app role assignment.


### Migrate to service principal

If your dbt Cloud app does not have a service principal, take the following actions in your Azure account:

1. Navigate to **Microsoft Entra ID**.
2. Under **Manage** on the left-side menu, click **App registrations**.
3. Click the app for the dbt Cloud and Azure DevOps integration.
4. Locate the **Managed application in local directory** field and click **Create Service Principal**.

## Connect a service user
<Lightbox src="/img/docs/cloud-integrations/create-service-principal.png" width="80%" title="Example of the 'Create Service Principal' option highlighted ."/>

Because Azure DevOps forces all authentication to be linked to a user's permissions, we recommend an Azure DevOps admin create a "service user" in Azure DevOps whose permissions will be used to power headless actions in dbt Cloud such as dbt Cloud project repo selection, deployment runs, and CI. A service user is a pseudo user set up in the same way an admin would set up a real user, but it's given permissions specifically scoped for service to service interactions. You should avoid linking authentication to a real Azure DevOps user because if this person leaves your organization, dbt Cloud will lose privileges to the dbt Azure DevOps repositories, causing production runs to fail.
5. Follow the instructions in [Add role to service principal](#add-a-role-to-the-service-principal) if the app doesn't already have them assigned.
6. In dbt Cloud, navigate to **Account settings** --> **Integrations** and edit the **Azure DevOps** integration.
7. Click the **Service principal** option, fill out the fields, and click **Save**. The services will continue to function uninterrupted.


</TabItem>

<TabItem value="Service user">

:::important

Service users are no longer a recommended method for authentication and accounts using them should [migrate](#migrate-to-service-principal) to Entra ID [service principals](https://learn.microsoft.com/en-us/entra/identity-platform/app-objects-and-service-principals) in the future. Service prinicpals are the [Microsoft recommended service account type](https://learn.microsoft.com/en-us/entra/architecture/secure-service-accounts#types-of-microsoft-entra-service-accounts) for app authentication.

:::

An Azure DevOps admin can create a "service user (not recommended)" in Azure DevOps whose permissions will be used to power headless actions in dbt Cloud such as dbt Cloud project repo selection, deployment runs, and CI. A service user is a pseudo user set up in the same way an admin would set up a real user, but it's given permissions specifically scoped for service to service interactions. You should avoid linking authentication to a real Azure DevOps user because if this person leaves your organization, dbt Cloud will lose privileges to the dbt Azure DevOps repositories, causing production runs to fail.

:::info Service user authentication expiration
dbt Cloud will refresh the authentication for the service user on each run triggered by the scheduler, API, or CI. If your account does not have any active runs for over 90 days, an admin will need to manually refresh the authentication of the service user by disconnecting and reconnecting the service user's profile via the OAuth flow described above in order to resume headless interactions like project set up, deployment runs, and CI.
Expand Down Expand Up @@ -393,3 +441,24 @@
- `vso.project`: Grants the ability to read projects and teams.
- `vso.build_execute`: Grants the ability to access build artifacts, including build results, definitions, and requests, and the ability to queue a build, update build properties, and the ability to receive notifications about build events with service hooks.
:::

</TabItem>
</Tabs>

## Add your Microsoft Entra ID app to dbt Cloud

A dbt Cloud account admin must take the following actions.

Once you connect your Microsoft Entra ID app and Azure DevOps, you need to provide dbt Cloud information about the app:

1. Navigate to your account settings in dbt Cloud.
2. Select **Integrations**.
3. Scroll to the Azure DevOps section.
4. Complete the form:
- **Azure DevOps Organization:** Must match the name of your Azure DevOps organization exactly. Do not include the `dev.azure.com/` prefix in this field. ✅ Use `my-devops-org` ❌ Avoid `dev.azure.com/my-devops-org`
- **Application (client) ID:** Found in the Microsoft Entra ID app.
- **Client Secrets:** Copy the **Value** field in the Microsoft Entra ID app client secrets and paste it in the **Client Secret** field in dbt Cloud. Entra ID admins are responsible for the Entra ID app secret expiration and dbt Admins should note the expiration date for rotation.
- **Directory(tenant) ID:** Found in the Microsoft Entra ID app.
<Lightbox src="/img/docs/cloud-integrations/service-principal-fields.png" title="Fields for adding Entra ID app to dbt Cloud."/>

Your Microsoft Entra ID app should now be added to your dbt Cloud Account. People on your team who want to develop in the dbt Cloud IDE or dbt Cloud CLI can now personally [authorize Azure DevOps from their profiles](/docs/cloud/git/authenticate-azure).
2 changes: 1 addition & 1 deletion website/docs/docs/dbt-versions/release-notes.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,12 @@ Release notes are grouped by month for both multi-tenant and virtual private clo

## December 2024

- **Enhancement**: [Service principal](/docs/cloud/git/setup-azure#configure-the-entra-id-connection) is now available as an authentication type for the Azure DevOps integration. Service principals offer a level of security and reliability above service users, so talk to your teams about migrating.
- **Fix**: [The dbt Semantic Layer](/docs/use-dbt-semantic-layer/dbt-sl) now respects the BigQuery [`execution_project` attribute](/docs/core/connect-data-platform/bigquery-setup#execution-project), including for exports.
- **New**: [Model notifications](/docs/deploy/model-notifications) are now generally available in dbt Cloud. These notifications alert model owners through email about any issues encountered by models and tests as soon as they occur while running a job.
- **New**: You can now use your [Azure OpenAI key](/docs/cloud/account-integrations?ai-integration=azure#ai-integrations) (available in beta) to use dbt Cloud features like [dbt Copilot](/docs/cloud/dbt-copilot) and [Ask dbt](/docs/cloud-integrations/snowflake-native-app) . Additionally, you can use your own [OpenAI API key](/docs/cloud/account-integrations?ai-integration=openai#ai-integrations) or use [dbt Labs-managed OpenAI](/docs/cloud/account-integrations?ai-integration=dbtlabs#ai-integrations) key. Refer to [AI integrations](/docs/cloud/account-integrations#ai-integrations) for more information.
- **New**: The [`hard_deletes`](/reference/resource-configs/hard-deletes) config gives you more control on how to handle deleted rows from the source. Supported options are `ignore` (default), `invalidate` (replaces the legacy `invalidate_hard_deletes=true`), and `new_record`. Note that `new_record` will create a new metadata column in the snapshot table.


## November 2024
- **Enhancement**: Trust signal icons in dbt Explorer are now available for Exposures, providing a quick view of data health while browsing resources. To view trust signal icons, go to dbt Explorer and click **Exposures** under the **Resource** tab. Refer to [Trust signal for resources](/docs/collaborate/explore-projects#trust-signals-for-resources) for more info.
- **Bug**: Identified and fixed an error with Semantic Layer queries that take longer than 10 minutes to complete.
Expand Down
Binary file added website/static/img/docs/cloud-integrations/assign-app-to-members.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/docs/cloud-integrations/azure-subscription.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/docs/cloud-integrations/create-service-principal.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/docs/cloud-integrations/review-and-assign.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added website/static/img/docs/cloud-integrations/service-principal-fields.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading