Bug: azurerm_kubernetes_cluster errors out with InvalidLoadbalancerProfile on changes

[aristosvo]

Fixes #6525

[aristosvo]

Before:

==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
TF_ACC=1 go test -v ./azurerm/internal/services/containers/tests/ -run=TestAccAzureRMKubernetesCluster_addAgent -timeout 60m -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccAzureRMKubernetesCluster_addAgent
=== PAUSE TestAccAzureRMKubernetesCluster_addAgent
=== CONT  TestAccAzureRMKubernetesCluster_addAgent
--- PASS: TestAccAzureRMKubernetesCluster_addAgent (1276.60s)
PASS
ok  	github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/containers/tests	1276.669s

After test enhancement:

==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
TF_ACC=1 go test -v ./azurerm/internal/services/containers/tests/ -run=TestAccAzureRMKubernetesCluster_addAgent -timeout 60m -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccAzureRMKubernetesCluster_addAgent
=== PAUSE TestAccAzureRMKubernetesCluster_addAgent
=== CONT  TestAccAzureRMKubernetesCluster_addAgent
--- FAIL: TestAccAzureRMKubernetesCluster_addAgent (1190.94s)
    testing.go:640: Step 2 error: errors during apply:
        
        Error: updating Managed Kubernetes Cluster "acctestaks200419171809471201" (Resource Group "acctestRG-200419171809471201"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.loadBalancerProfile"
        
          on /tmp/tf-test124628542/main.tf line 21:
          (source code not available)
        
        
FAIL
FAIL	github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/containers/tests	1190.991s
FAIL
GNUmakefile:100: recipe for target 'acctests' failed

After code change:

==> Checking that code complies with gofmt requirements...
==> Checking that Custom Timeouts are used...
TF_ACC=1 go test -v ./azurerm/internal/services/containers/tests/ -run=TestAccAzureRMKubernetesCluster_addAgent -timeout 60m -ldflags="-X=github.com/terraform-providers/terraform-provider-azurerm/version.ProviderVersion=acc"
=== RUN   TestAccAzureRMKubernetesCluster_addAgent
=== PAUSE TestAccAzureRMKubernetesCluster_addAgent
=== CONT  TestAccAzureRMKubernetesCluster_addAgent
--- PASS: TestAccAzureRMKubernetesCluster_addAgent (1507.55s)
PASS
ok  	github.com/terraform-providers/terraform-provider-azurerm/azurerm/internal/services/containers/tests	1507.635s

[sbkg0002]

This would really help us! Is there anything we can do to help the merge?

[AntonChernysh]

Another one is looking forward for this PR to be merged.
...
kubelet_identity is not exposed in 2.5.0 and we really need 2.6.0 or above but can't do it due to this bug.

[leiter-jakab]

@AntonChernysh

I used this to expose the kubelet identity before 2.6:

data "azuread_service_principal" "node_identity" {
  display_name = "${azurerm_kubernetes_cluster.my_cluster.name}-agentpool"
}

output "node_service_principal_id" {
  value = data.azuread_service_principal.node_identity.id
}

[angelbarrera92]

We are facing this issue. Is this PR planned to be merged any time soon?

[ekarlso]

@tombuildsstuff Any idea on getting this in for a quick 2.8.0 or 2.7.1 ? It's a really sad issue.

[twendt]

I have just successfully tested this PR.

[sbkg0002]

It's not part of the 2.9 release, is it?

[aristosvo]

Nope. I'll rebase it to resolve the conflicts

[gitflo1]

Will this PR be part of the 2.10 release? This is really blocking us from updating our clusters...

[aristosvo]

I discussed it with the maintainers, if it doesn’t make it into 2.10.0 (May 14th) expect it for 2.11.0.

[atrauzzi]

Can we please have this in a patch release 2.9.1, ASAP? This bug is far too disruptive to sit on a fix for the regular release cycle.

[djsly]

this needs to be picked up ASAP :) we will be reverting back to a custom local provider code, but we hate doing this... @aristosvo any feedback on your PR from @tombuildsstuff / @katbyte yet ? I quickly reviewed , and it seems good.

[aristosvo]

@djsly this:

if it doesn’t make it into 2.10.0 expect it for 2.11.0.

@katbyte has taken a look at the PR and told me she will discuss it internally with someone more familiar with AKS, probably @tombuildsstuff. One of the main reasons they are hesitant to merge AKS PRs at the moment is due to a lot of breaking API changes for AKS lately..

[djsly]

@aristosvo I understand, so many changes. but this not being merged is breaking the usage of terraform + aks

[katbyte]

Thanks for this fix @aristosvo, LGTM 👍

[ghost]

This has been released in version 2.10.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.10.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!