Error updating VmwareCluster

[richardmoe]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request.
• Please do not leave +1 or me too comments, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.
• If an issue is assigned to a user, that user is claiming responsibility for the issue.
• Customers working with a Google Technical Account Manager or Customer Engineer can ask them to reach out internally to expedite investigation and resolution of this issue.

Terraform Version

Terraform v1.8.0
on linux_amd64

• provider registry.terraform.io/akamai/akamai v6.0.0
• provider registry.terraform.io/grafana/grafana v2.18.0
• provider registry.terraform.io/hashicorp/azuread v2.48.0
• provider registry.terraform.io/hashicorp/azurerm v3.100.0
• provider registry.terraform.io/hashicorp/external v2.3.3
• provider registry.terraform.io/hashicorp/google v5.25.0
• provider registry.terraform.io/hashicorp/helm v2.12.1
• provider registry.terraform.io/hashicorp/kubernetes v2.29.0
• provider registry.terraform.io/hashicorp/null v3.2.2
• provider registry.terraform.io/hashicorp/random v3.6.1
• provider registry.terraform.io/hashicorp/time v0.11.1
• provider registry.terraform.io/hashicorp/tls v4.0.5
• provider registry.terraform.io/hashicorp/vault v4.2.0
• provider registry.terraform.io/nrkno/lastpass v0.5.3
• provider registry.terraform.io/tenstad/remote v0.1.2

Affected Resource(s)

google_gkeonprem_vmware_cluster

Terraform Configuration

resource "google_gkeonprem_vmware_cluster" "gkee_onprem_cluster" {
  description              = "Onprem cluster deployed in VmWare by Terraform"
  admin_cluster_membership = var.cluster_admin_cluster_membership
  on_prem_version          = var.cluster_on_prem_version
  name                     = var.cluster_name
  location                 = var.cluster_location

Debug Output

https://gist.github.com/richardmoe/10698bf3fb147effa489b67235d1f727

Expected Behavior

Upgraded to specified version

Actual Behavior

Provider crash

Steps to reproduce

1. terraform apply

Important Factoids

No response

References

No response

b/339236238

[zli82016]

Hello, @richardmoe , can you please provide more information to help reproduce the issue? Thanks.

1. The version of Terraform Google provider
2. The complete configuration before and after updating
3. The DEBUG logs when running the command TF_LOG=DEBUG terraform apply. The current logs don't have so much information.

[richardmoe]

1. The provider version is listed above: registry.terraform.io/hashicorp/google v5.25.0, but also tested with v5.26.0
2. TF config is here: https://gist.github.com/richardmoe/5d232059fa74bf793d28c9050487ffeb. The only change is image = "gke-on-prem-ubuntu-1.28.300-gke.123" -> "gke-on-prem-ubuntu-1.28.400-gke.75"
3. TF LOG is here. Only WARN level, since DEBUG contains secrets: https://gist.github.com/richardmoe/5428ff722283f25a0d08dc92156d0a9e

[zli82016]

Forwarding to the service team to check the error

│ Error: Error updating VmwareCluster "projects/***": googleapi: Error 400: Proto is invalid: [version_update: control plane upgrade disallowed with cluster config updates]

[NickElliot]

hi @richardmoe, we're having a hard time reproducing this. Would it be possible for you to share a scrubbed version of the PATCH request within the debug logs?

[naitianliu-google]

Hi @richardmoe, the error returned means it is not allowed the modify other fields such as image as mentioned when modifying the on_prem_version field. Could you please help confirm if only on_prem_version is modified to perform the upgrade operation?

[richardmoe]

Here is the relevant PATCH request @NickElliot

2024-05-08T11:48:30.603+0200 [DEBUG] module.gkee-plattform-lab-onprem.google_gkeonprem_vmware_cluster.gkee_onprem_cluster: applying the planned Update change
2024-05-08T11:48:30.611+0200 [DEBUG] provider.terraform-provider-google_v5.28.0_x5: 2024/05/08 11:48:30 [DEBUG] Updating VmwareCluster "projects/nrk-platt-gke-enterprise-test/locations/europe-west3/vmwareClusters/gkee-plattform-lab": map[string]interface {}{"annotations":map[string]string{"onprem.cluster.gke.io/block-docker-node-pools":"true", "onprem.cluster.gke.io/error-reason":"", "onprem.cluster.gke.io/gke-on-prem-last-update-start-time":"2024-04-29T10:29:09Z", "onprem.cluster.gke.io/operation":"upgrade", "onprem.cluster.gke.io/operation-id":"f8bcfef4-b117-4e22-8e2f-c9e2e448d614", "onprem.cluster.gke.io/source":"api-vmware", "onprem.cluster.gke.io/start-time":"2024-04-29T11:26:54Z", "onprem.cluster.gke.io/status":"started", "onprem.cluster.gke.io/user-cluster-resource-link":"//gkeonprem.googleapis.com/projects/464094393805/locations/europe-west3/vmwareClusters/gkee-plattform-lab"}, "antiAffinityGroups":map[string]interface {}{}, "authorization":map[string]interface {}{"adminUsers":[]interface {}{map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"Vidar.Thomassen.redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}, map[string]interface {}{"username":"redacted@nrk.no"}}}, "autoRepairConfig":map[string]interface {}{"enabled":true}, "controlPlaneNode":map[string]interface {}{"autoResizeConfig":map[string]interface {}{"enabled":true}, "cpus":4, "memory":8196, "replicas":3, "vsphereConfig":map[string]interface {}{"datastore":"IPLAB_NFSDATASTORE"}}, "dataplaneV2":map[string]interface {}{"advancedNetworking":true, "dataplaneV2Enabled":true, "windowsDataplaneV2Enabled":true}, "description":"Onprem cluster deployed in VmWare by Terraform", "disableBundledIngress":true, "enableControlPlaneV2":true, "loadBalancer":map[string]interface {}{"metalLbConfig":map[string]interface {}{"addressPools":[]interface {}{map[string]interface {}{"addresses":[]interface {}{"10.252.242.98-10.252.242.126"}, "avoidBuggyIps":true, "pool":"gke-usercluster1-lab-lb-pool"}}}, "vipConfig":map[string]interface {}{"controlPlaneVip":"10.252.242.97"}}, "networkConfig":map[string]interface {}{"controlPlaneV2Config":map[string]interface {}{"controlPlaneIpBlock":map[string]interface {}{"gateway":"10.252.242.1", "ips":[]interface {}{map[string]interface {}{"hostname":"gkee-plattform-lab-control-plane-01", "ip":"10.252.242.5"}, map[string]interface {}{"hostname":"gkee-plattform-lab-control-plane-02", "ip":"10.252.242.6"}, map[string]interface {}{"hostname":"gkee-plattform-lab-control-plane-03", "ip":"10.252.242.7"}}, "netmask":"255.255.255.128"}}, "dhcpIpConfig":map[string]interface {}{"enabled":true}, "hostConfig":map[string]interface {}{"dnsServers":[]interface {}{"10.252.243.2", "10.252.243.3"}, "ntpServers":[]interface {}{"ntp.uio.no"}}, "podAddressCidrBlocks":[]interface {}{"10.124.0.0/16"}, "serviceAddressCidrBlocks":[]interface {}{"192.168.20.0/22"}, "vcenterNetwork":"IP-LAB/gke-usercluster1-lab"}, "onPremVersion":"1.28.400-gke.75", "storage":map[string]interface {}{}, "upgradePolicy":map[string]interface {}{}, "vcenter":map[string]interface {}{"address":"malabvc01.vmware.nrk.cloud", "caCertData":"redacted, "cluster":"IP-LAB", "datacenter":"Marienlyst", "datastore":"IPLAB_NFSDATASTORE", "folder":"/Marienlyst/vm/IP-LAB", "resourcePool":"IP-LAB/Resources"}}
2024-05-08T11:48:30.612+0200 [DEBUG] provider.terraform-provider-google_v5.28.0_x5: PATCH /v1/projects/nrk-platt-gke-enterprise-test/locations/europe-west3/vmwareClusters/gkee-plattform-lab?alt=json&updateMask=onPremVersion%2CantiAffinityGroups%2Cstorage%2CupgradePolicy HTTP/1.1
2024-05-08T11:48:30.612+0200 [DEBUG] provider.terraform-provider-google_v5.28.0_x5:   "onprem.cluster.gke.io/source": "api-vmware",
2024-05-08T11:48:30.612+0200 [DEBUG] provider.terraform-provider-google_v5.28.0_x5:   "onprem.cluster.gke.io/user-cluster-resource-link": "//gkeonprem.googleapis.com/projects/464094393805/locations/europe-west3/vmwareClusters/gkee-plattform-lab"
2024-05-08T11:48:30.613+0200 [DEBUG] provider.terraform-provider-google_v5.28.0_x5:   "address": "malabvc01.vmware.nrk.cloud",
2024-05-08T11:48:31.276+0200 [ERROR] provider.terraform-provider-google_v5.28.0_x5: Response contains error diagnostic: diagnostic_detail="" diagnostic_severity=ERROR tf_proto_version=5.4 tf_provider_addr=registry.terraform.io/hashicorp/google tf_req_id=7ddb4cd7-eeb7-af50-9eb5-2f670c49bab6 @caller=github.com/hashicorp/terraform-plugin-go@v0.22.1/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto tf_resource_type=google_gkeonprem_vmware_cluster tf_rpc=ApplyResourceChange
  | Error updating VmwareCluster "projects/nrk-platt-gke-enterprise-test/locations/europe-west3/vmwareClusters/gkee-plattform-lab": googleapi: Error 400: Proto is invalid: [version_update: control plane upgrade disallowed with cluster config updates]
2024-05-08T11:48:31.286+0200 [ERROR] vertex "module.gkee-plattform-lab-onprem.google_gkeonprem_vmware_cluster.gkee_onprem_cluster" error: Error updating VmwareCluster "projects/nrk-platt-gke-enterprise-test/locations/europe-west3/vmwareClusters/gkee-plattform-lab": googleapi: Error 400: Proto is invalid: [version_update: control plane upgrade disallowed with cluster config updates]``` 

[NickElliot]

could you share your terraform show output for the resource youre attempting to update, where specifically i want to know what it is showing as your read state for onPremVersion, antiAffinityGroups{}, storage{}, and upgradePolicy{}

[richardmoe]

output of terraform show: https://gist.github.com/richardmoe/a05dbb8f000bffbb5de57f0396a9092c

[NickElliot]

@naitianliu-google the cause of this issue is the surrounding objects of boolean fields set to false are being added to the update mask of a request because those objects mentioned in #17933 (comment) are included in the patch request as empty structs. This should be resolvable by adjusting the resource to not add these empty structs to the update mask, or not send the empty structs within each post request at all (if the API would allow for this).

Thanks for the detailed diagnostic info @richardmoe!