InvalidLoadBalancerProfile in AKS

[MikeMichel]

After deplyoing an app with a load balancer into a fresh AKS cluster with no load balancer profile configured (only sku to Standard) and then changing for example number of nodes gives me:

Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must
specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." 
Target="networkProfile.loadBalancerProfile"

Output of terrafom apply which is causing it I guess:

            load_balancer_profile {
                effective_outbound_ips    = [
                    "/subscriptions/..../publicIPAddresses/.....",
                ]
                managed_outbound_ip_count = 1
                outbound_ip_address_ids   = []
                outbound_ip_prefix_ids    = []
            }
        }

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.23

• provider.azurerm v2.6.0

• #0000

[ablyler]

I am having the same issue after upgrading to azure provider v2.6.0. I have tried to fix the issue by adding the load_balancer_profile block as so:

  network_profile {
    network_plugin    = "azure"
    load_balancer_sku = "standard"
    load_balancer_profile {
      managed_outbound_ip_count = 1
    }
  }

However, I get the same error in the description of this issue.

[asubmani]

I am getting the same error. I am using terraform v0.12.24 and using azurerm provider v2.6.0

Provisioned the AKS cluster successfully.
Terraform Plan about "Changing" SPN for AKS RBAC. # I did not make any changes to SPN details as my cluster is using Managed Identities.

I get the error only during terraform apply

module.AKSClus.azurerm_kubernetes_cluster.TerraAKSwithRBAC: Still modifying
... [id=/subscriptions/xxxx-1193-4xxx-xxxxx7-...rService/managedClusters/
jb-aks-eus-qa, 1m0s elapsed]

Error: updating Managed Kubernetes Cluster "jb-aks-eus-qa" (Resource Group
"xx-rg-aks-qa"): containerservice.ManagedClustersClient#CreateOrUpdate: Fai
lure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBa
lancerProfile" Message="Load balancer profile must specify one of ManagedOu
tboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.load
BalancerProfile"

[aristosvo]

Ah, shoot! I just pushed a fix PR.

I've spent some thoughts on this scenario during implementation of the feature causing this issue, but didn't implement the right test for it and forgot!

[gitflo1]

Getting the same error as described above for the following scenario, using terraform version v.0.12.24 while upgrading from AKS version 0.15.7 to 0.16.7 with the following config:

  network_profile {
    network_plugin     = "azure"
    network_policy     = "azure"
    dns_service_ip     = var.aks_dns_service_ip
    docker_bridge_cidr = var.aks_docker_bridge_cidr
    service_cidr       = var.aks_service_cidr
    outbound_type = "loadBalancer"

    // For egress configuration
    load_balancer_sku  = "Standard"
    load_balancer_profile {
      outbound_ip_address_ids = [ var.aks_egress_ip_id ]
    }
  }

Terraform tries an update in place and fails with the error message displayed in the original post.

[bytebounder]

As an interim measure, I rolled back my azurerm provider version to 2.5.0.

[aristosvo]

@tombuildsstuff Can we tag this as a /bug? The fix is already available, including a test reproducing the failure.

@robologic et all: as a workaround you can also change the number of outbound IP's for every change, the problem shows when no change in the load_balancer_profile is applied. In that case the load_balancer_profile configuration is not included in the request towards Azure 😢

[tezzigator]

As an interim measure, I rolled back my azurerm provider version to 2.5.0.

This doesnt work for me as I need 2.6.0 for the kubelet exports.

I believe I have the same issue here on 2.6.0; I can create the AKS in Devops no problems, but if later then I try to change on that AKS with terraform apply via Devops for any change on that AKS, it doesnt have to be LB or IP related, for example, changing the IP blocks allowed into the K8S API, then Devops fails with

2020-04-22T03:33:15.8987559Z �[1m�[31mError: �[0m�[0m�[1mupdating Managed Kubernetes Cluster "REDACTED" (Resource Group "REDACTED"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.loadBalancerProfile"�[0m
2020-04-22T03:33:15.8989767Z 
2020-04-22T03:33:15.8990917Z �[0m  on modules/aks/main.tf line 7, in resource "azurerm_kubernetes_cluster" "aks":
2020-04-22T03:33:15.8992918Z    7: resource "azurerm_kubernetes_cluster" "aks" �[4m{�[0m
2020-04-22T03:33:15.8993692Z �[0m
2020-04-22T03:33:15.8994180Z �[0m�[0m
2020-04-22T03:33:16.0389462Z ##[error]Error: The process '/opt/hostedtoolcache/terraform/0.12.24/x64/terraform' failed with exit code 1

[chrishatfield86]

I'm seeing the same behaviour on 2.6.0 and don't wish to roll back at this stage. I can initially provision a cluster fine with a load_balancer_profile block but any subsequent terraform apply will fail. These subsequent applies having nothing in the plan indicating any change required for the network_profile and load_balancer_profile

##[error]Terraform command 'apply' failed with exit code '1'.:  updating Managed Kubernetes Cluster "my_cluster" (Resource Group "my_rg"): containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." Target="networkProfile.loadBalancerProfile"

[titilambert]

Hello,
I just tested the master branch (v2.6 ++ ) and it worked !
Thanks @aristosvo

[asubmani]

I am still getting this error with provider 2.7.0. I have to keep changing managed_outbound_ip_count = 1 to a different number each time I run my plan & apply.
The plan does not catch this (unless I change the value); but the terraform apply command throws this error.

[aristosvo]

@asubmani I'm sorry, it is not merged yet. If any of you can thoroughly review #6534 it might speed up things a bit, but other than that I can do nothing.

👍 on the issue is the best way for the maintainers to indicate the need for the fix.

[jmcshane]

I'm still having an issue now on 2.7.0, my load balancer profile just sets managed_outbound_ip_count but it is failing with the same message.

[asubmani]

@asubmani I'm sorry, it is not merged yet. If any of you can thoroughly review #6534 it might speed up things a bit, but other than that I can do nothing.

👍 on the issue is the best way for the maintainers to indicate the need for the fix.

Thank you... so I fork your repo from #6534 and build it using make for golang? I am not a developer, but if you could point me to some basic doc I can do my bit.

[angelbarrera92]

This was working in 2.5.0

[aristosvo]

It is still working in 2.5.0!

To make it possible to change your loadbalancer outbound IP settings without recreating the cluster I filled a PR, which is merged into 2.6.0. Due to this PR the loadbalancer outbound settings can be changed. But if these settings are not changed a bug in the code causes your AKS cluster to think you have no outbound loadbalancer settings, which blocks any other kind of change you want to make to your cluster.

The good news is 2.5.0 is still working and just limited functionality has made it past that. If you need 2.6+ functionality, this sad issue is blocking cluster changes after cluster creation.

[jmcshane]

So does this mean that clusters created with the provider before 2.5 are not able to be modified if the provider version is bumped above 2.6? Is there an ignore value that we can set?

[markslater]

This issue (in my experience), means that all versions of the provider > 2.5 are broken for AKS, regardless of the version of the provider used to create the AKS cluster.

It's actually a rather insidious break, because you can create AKS clusters using newer versions of the provider, but the first time you have a modification, terraform apply will crap out every time.

To me, this appears to be a grade A drop-everything-and-fix-it issue because to all intents and purposes the past four provider releases are fundamentally broken for all AKS users... please excuse me if I've got the wrong end of the stick and there's a workaround I missed!

[aristosvo]

I discussed it with the maintainers, if it doesn’t make it into 2.10.0 (May 14th) expect it for 2.11.0.

[atrauzzi]

Just encountered this. This is a pretty nasty issue and I have to agree with @markslater, this needs to be fixed and released sooner than a regular release cycle.

Can we ping the maintainers in here for discussion? Do they ascertain the severity of this or can they provide sound reasoning?

[djsly]

is this PR the official one ? #6534. we need a custom build as this is breaking all cluster update :(

[viresh-contino]

I have just observed this with 2.9.0.
this is the error message:
containerservice.ManagedClustersClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must specify one of ManagedOutboundIPs

I basically did a terraform plan, terraform apply and then another terraform plan and terraform apply.

[ghost]

This has been released in version 2.10.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.10.0"
}
# ... other configuration ...

[atrauzzi]

Just linked in the issue above. If you're looking to upgrade and are using the Basic SKU of postgres, you'll encounter another show-stopper bug.

[aristosvo]

It’s quit easy to separate the versions between two resources, although I feel your pain.

Bugs like these on advanced resources are not always easy to catch though..

[atrauzzi]

To be honest, I don't feel like I've been doing anything advanced. I've hit so many bugs and gotchas just trying to provision a resource group with a database, gateway, AKS and key store.

All low tier and lots of defaults.

[nivhty]

I'm having the same issue as well on:

• Terraform v0.12.24
• Azurerm provider 2.8.0
  When I do a re-apply after adding a tags block in the code, I was getting the same error message:

Error: Code="InvalidLoadBalancerProfile" Message="Load balancer profile must
specify one of ManagedOutboundIPs, OutboundIPPrefixes and OutboundIPs." 
Target="networkProfile.loadBalancerProfile"

My network_profile block is below:

  network_profile {
    network_plugin     = var.cluster_network_plugin
    dns_service_ip     = var.cluster_dns_service_ip
    docker_bridge_cidr = var.cluster_docker_bridge_cidr
    service_cidr       = var.cluster_service_cidr

    # Standard LoadBalancer is required for multiple nodepools.
    load_balancer_sku = "standard"

    # Use a single IP for outbound traffic
    load_balancer_profile {
      outbound_ip_address_ids = [azurerm_public_ip.outbound.id]
    }
  }

[aristosvo]

I'm having the same issue as well on:

• Terraform v0.12.24

• Azurerm provider 2.8.0

@nivhty upgrade to 2.10.0 for the fix.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!