Azurerm provider unable to update Azure App Gateway when using ssl certificates from key vault

[kirangurumukhi]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

Terraform v1.7.3 on linux_amd64

AzureRM Provider Version

3.46.0

Affected Resource(s)/Data Source(s)

azurerm_application_gateway

Terraform Configuration Files

# var.deployed_customer_names is a map containing customer_name and product types to install

resource "azurerm_application_gateway" "appgateway" {
  name                         = "agw-${var.product_app_code}-${var.env_code}-${var.azure_region_shortname}-${var.application}"
  location                     = azurerm_resource_group.resource_group.location
  resource_group_name          = azurerm_resource_group.resource_group.name
  tags                         = merge(var.tags, tomap(jsondecode("{\"created_on\": \"${formatdate("MM-DD-YYYY", timestamp())}\"}")))
  sku {
    name     = "Standard_v2"
    tier     = "Standard_v2"
    capacity = 2
  }

  ssl_policy {
    policy_type             = "Predefined"
    min_protocol_version    = "TLSv1_2"
    policy_name = "AppGwSslPolicy20220101"
  }

  gateway_ip_configuration {
    name      = "appgwy-ip-config"
    subnet_id = azurerm_subnet.waf_subnet.id
  }


  dynamic "ssl_certificate" {
    for_each = var.deployed_customer_names
    content {
      name                        = ssl_certificate.value.customer_name
      key_vault_secret_id         = data.azurerm_key_vault_certificate.app-gwy-certs[ssl_certificate.key].secret_id
    }
  }

  frontend_ip_configuration {
    name                            = "Public"
    public_ip_address_id            = azurerm_public_ip.Ingress_IP.id
  }
  
  frontend_port {
    name = "Port_443"
    port = 443
  }


  dynamic "backend_address_pool" {
    for_each = local.flat-customers

    content {
      name         = "${backend_address_pool.value.customer_name}-${backend_address_pool.value.product_type}"
      fqdns        = ["${backend_address_pool.value.customer_name}-${backend_address_pool.value.product_type}.${azurerm_private_dns_zone.privatelink_deployments.name}"]
    }
  }

  # dynamic "backend_address_pool" {
  #   for_each = var.deployed_customer_names

  #   content {
  #     name         = "${backend_address_pool.value.customer_name}-${backend_address_pool.value.product_type}"
  #     fqdns        = ["${backend_address_pool.value.customer_name}-${backend_address_pool.value.product_type}.${azurerm_private_dns_zone.privatelink_deployments.name}"]
  #   }
  # }

  dynamic "backend_http_settings" {
    for_each =  local.flat-customers
    
    content {
      name                  = "${backend_http_settings.value.customer_name}-${backend_http_settings.value.product_type}"
      cookie_based_affinity = "Disabled"
      path                  = backend_http_settings.value.product_type == "platform" ? "" : "/"
      port                  = 80
      protocol              = "Http"
      request_timeout       = 30
      probe_name            = "${backend_http_settings.value.customer_name}-${backend_http_settings.value.product_type}"
    }
  }

  dynamic "probe" {
    for_each =  local.flat-customers

    content {
      host                                      = "${probe.value.customer_name}-${probe.value.product_type}.${var.domain_postfix}"
      name                                      = "${probe.value.customer_name}-${probe.value.product_type}"
      path                                      = "/healthz"
      interval                                  = 30
      protocol                                  = "Http"
      unhealthy_threshold                       = 8
      timeout                                   = 30
    }
  }

  dynamic "http_listener" {
   for_each =  var.deployed_customer_names

    content {
      name                           = http_listener.value.customer_name
      frontend_ip_configuration_name = "Public"
      frontend_port_name             = "Port_443"
      protocol                       = "Https"
      require_sni                    = false
      ssl_certificate_name           = http_listener.value.customer_name
      host_name                      = "${http_listener.value.customer_name}.${var.domain_postfix}"
    }
  }

  dynamic "request_routing_rule" {
    for_each =  var.deployed_customer_names
    
    content {
      name                        = request_routing_rule.value.customer_name
      priority                    = index(keys(var.deployed_customer_names), request_routing_rule.key)+1
      rule_type                   = "PathBasedRouting"
      http_listener_name          = request_routing_rule.value.customer_name
      # backend_address_pool_name   = request_routing_rule.value
      # backend_http_settings_name  = request_routing_rule.value
      url_path_map_name           = request_routing_rule.value.customer_name
    }
  }

  dynamic url_path_map {
    for_each =  var.deployed_customer_names

    content {
      name                                = url_path_map.value.customer_name
      default_backend_address_pool_name   = "${url_path_map.value.customer_name}-${url_path_map.value.product_types[0]}"
      default_backend_http_settings_name  = "${url_path_map.value.customer_name}-${url_path_map.value.product_types[0]}"

      dynamic "path_rule" {
        for_each =  toset(url_path_map.value.product_types)
        content {
          name                        = path_rule.value
          paths                       = path_rule.value == "platform" ? ["/*"] : ["/${path_rule.value}/*"]
          backend_address_pool_name   = "${url_path_map.value.customer_name}-${path_rule.value}"
          backend_http_settings_name  = "${url_path_map.value.customer_name}-${path_rule.value}"
        }
      }
    }
  }


  identity {
    type = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.keyvault_identity.id]
  }

  depends_on = [ azurerm_key_vault_certificate.placeholder_certificate ]
}

Debug Output/Panic Output

Command failure. 
Error: Provider produced inconsistent final plan

When expanding the plan for azurerm_application_gateway.appgateway to include
new values learned so far during apply, provider
"registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
.ssl_certificate: planned set element
cty.ObjectVal(map[string]cty.Value{"data":cty.StringVal(""),
"id":cty.UnknownVal(cty.String),
"key_vault_secret_id":cty.UnknownVal(cty.String),
"name":cty.StringVal("essdemo"), "password":cty.StringVal(""),
"public_cert_data":cty.UnknownVal(cty.String)}) does not correlate with any
element in actual.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.
Command failure. 
Error: Provider produced inconsistent final plan

When expanding the plan for azurerm_application_gateway.appgateway to include
new values learned so far during apply, provider
"registry.terraform.io/hashicorp/azurerm" produced an invalid new value for
.ssl_certificate: planned set element
cty.ObjectVal(map[string]cty.Value{"data":cty.StringVal(""),
"id":cty.UnknownVal(cty.String),
"key_vault_secret_id":cty.UnknownVal(cty.String),
"name":cty.StringVal("essdemo"), "password":cty.StringVal(""),
"public_cert_data":cty.UnknownVal(cty.String)}) does not correlate with any
element in actual.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Expected Behaviour

Should have updated the application gateway based on the listener configuration, reading the certs from the key vault.

Actual Behaviour

Gives an error as indicated in the output. Interestingly, if the SSL certificates in the Azure key vault are just updated with new version (uploading the old certificate itself), the error goes away.

Steps to Reproduce

Entry criteria - Azure Application Gateway is deployed via terraform with just one customer name. At this time, there is one listener, mapped to SSL certificate reading from the key vault. No issue till this step. TF runs and updates the App gateway appropriately.

Add another customer to the var.deployed_customer_names, ensuring that the key vault also has the certificate for this new customer. If you run terraform now, it errors out stating that TF generated inconsistent plan. What should have happened is that the previous listener should have been kept on the gateway (since there are no changes) and the new listener should have been added. However, what happens is that TF somehow generates invalid values or plan and errors out.

Important Factoids

Azure running in East US Region.

References

No response

[teowa]

@kirangurumukhi , thanks for submitting this!
I found this might be related with changes introduced in #8761

terraform-provider-azurerm/internal/services/network/application_gateway_resource.go

Lines 4101 to 4102 in 6440011

	} else if cert != "" {
	output.ApplicationGatewaySslCertificatePropertiesFormat.PublicCertData = utils.String(cert)

the public_cert_data is computed but is expanded. Let me do more tests and then try to fix it.