Error while encrypting Azure Databricks Cluster Disks using CMK

[abhinavkumar7]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.8.2

AzureRM Provider Version

3.99.0

Affected Resource(s)/Data Source(s)

azurerm_databricks_workspace

Terraform Configuration Files

resource "azurerm_databricks_workspace" "example" {
  name                = "databricks-test"
  resource_group_name = azurerm_resource_group.example.name
  location            = azurerm_resource_group.example.location
  sku                 = "standard"
  managed_disk_cmk_key_vault_key_id                   = azurerm_key_vault_key.cmk_kv.id
  managed_disk_cmk_rotation_to_latest_version_enabled = "true"

  tags = {
    Environment = "Production"
  }
}

Debug Output/Panic Output

Error: creating/updating Workspace (Subscription: "***"
Resource Group Name: "test-rg"
Workspace Name: "test-dbw"): polling after CreateOrUpdate: polling failed: the Azure API returned the following error:

Status: "KeyVaultAccessForbidden"
Code: ""
Message: "Unable to perform update operation for the workspace : '/subscriptions/***/resourceGroups/test-rg/providers/Microsoft.Databricks/workspaces/test-dbw' failed : 'Unable to access key vault 'https://my-test-cmk.vault.azure.net/keys/cmk-test''."
Activity Id: ""

---

API Response:

----[start]----
{
  "status": "Failed",
  "error": {
    "code": "KeyVaultAccessForbidden",
    "message": "Unable to perform update operation for the workspace : '/subscriptions/***/resourceGroups/test-rg/providers/Microsoft.Databricks/workspaces/test-dbw' failed : 'Unable to access key vault 'https://my-test-cmk.vault.azure.net/keys/cmk-test''."
  }
}
-----[end]-----


  with azurerm_databricks_workspace.this,
  on databricks.tf line 1, in resource "azurerm_databricks_workspace" "this":
   1: resource "azurerm_databricks_workspace" "this" {

Releasing state lock. This may take a few moments...

Expected Behaviour

There should be a way to grant access to the principal of automatically created DES resource on the CMK Key vault, while enabling CMK for Azure Databricks disks. So that the CMK encryption of Azure Databricks disk works correctly.

Actual Behaviour

When I enable CMK for Disks in Azure Databricks, a DES resource is automatically created in Azure and CMK encryption fails for the disks, since the identity of DES resource doesn't have access to the CMK KV.

Steps to Reproduce

Try to enable CMK for Azure Databricks cluster node disks.

Important Factoids

No response

References

No response