Databricks workspace not ready to use after creation

[sdebruyn]

API endpoints:

• https://docs.microsoft.com/en-us/rest/api/databricks/workspaces/createorupdate
• https://docs.microsoft.com/en-us/rest/api/databricks/workspaces/get

When you create a Databricks workspace and then try to create a cluster, you get an error saying that the worker environment is not ready yet.

The response includes a field called provisioningState. So far, I've only seen the succeeded state. The error occurs in the first few seconds right after the workspace is created, but the state doesn't change when the error goes away.

When you try to create a cluster, you get HTTP 400 with the following message: Current organization XXXX does not have any associated worker environments

For consistency, please change the state according to its availability.

First reported on:

• Databricks workspace not ready after creation hashicorp/terraform-provider-azurerm#6753
• https://github.com/innovationnorway/terraform-provider-databricks/issues/49
• [ISSUE] UnknownWorkerEnvironmentException when creating a cluster after creating a workspace databricks/terraform-provider-databricks#33

[ghost]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @yagupta.

[mssaperla]

@sdebruyn - Thank you for your question and apologize for the late response. We applied some optimizations to this workflow in July. Can you comment on whether you are still observing this issue? @mvvsubbu will work with you on this issue.

[mvvsubbu]

@sdebruyn

We apologize for the delay in responding for the issue. I want to give you some context on the workspace provisioning state and cluster state. These two are different states which indicate different things. Workspace provisioning state shouldn't be updated to Failed if one cluster provisioning fails, as there can be more than one cluster possible in a workspace. So the state of the workspace succeeded is correct and that shouldn't be updated based on cluster status.

Secondly, the cluster creation failure if you create immediately after workspace creation. This flow we have done some improvements in July and that should improve your reliability and not see this issue. Please let us know if you are still seeing this issue, we can work on understanding and addressing.

If my response addresses your questions, feel free to close the issue.

[mvvsubbu]

@sdebruyn I am closing the issue for now, please feel free to reopen if you are still observing issues with creation of cluster after workspace creation.