[FEATURE] Support Azure RBAC on Azure Key Vault backed secret scopes

[audunsolemdal]

Problem Statement

Same issue as databricks/terraform-provider-databricks#1206
Support Azure RBAC on key vaults.

My understanding is that the current solution automatically creates an azure key vault access policy and grants access to the AzureDatabricks enterprise app, and this can then be used via dbutils to read secrets into e.g. notebooks.

The recommended way to authenticate towards Azure key vaults is by using Azure RBAC, not using Key Vault access policies. For my tenant the policy is to use Azure RBAC, and vaults connected using databricks secret scopes are the only exclusions we currently have to this policy.

Proposed Solution

The "Azure Databricks" enterprise app

Should be granted the Key Vault Secrets User Azure RBAC role. This should be sufficient to authenticate secret scopes towards azure key vaults.

This can be granted by a user or service principal with one of the following roles

Key Vault Data Access Administrator (preffered)
User Acesss Administrator
Owner

Additional Context

https://learn.microsoft.com/en-us/azure/databricks/security/secrets/#configure-your-azure-key-vault-instance-for-azure-databricks