Is it intentional that order of metrics matter on HPAs? #1188

marjorg · 2021-03-08T14:44:19Z

Terraform Version, Provider Version and Kubernetes Version

Terraform version: v0.14.7
Kubernetes provider version: v2.0.2
Kubernetes version:  v1.19

Affected Resource(s)

kubernetes_horizontal_pod_autoscaler
kubernetes_service
kubernetes_deployment

Using this with AWS, not sure if that matters.

Terraform Configuration Files

resource "kubernetes_deployment" "my_app_deployment" {
	metadata {
		name = "my-app-deployment"
		labels = {
			app = "my-app"
		}
	}

	spec {
		revision_history_limit = 3

		selector {
			match_labels = {
				app = "my-app"
			}
		}

		template {
			metadata {
				labels = {
					app = "my-app"
				}
			}

			spec {
				container {
					name = "my-app"
					image = "IMAGE_URL"
					image_pull_policy = "Always"

					port {
						container_port = 80
					}

					resources {
						requests = {
							cpu = "200m"
							memory = "200Mi"
						}
						limits = {
							cpu    = "600m"
							memory = "600Mi"
						}
					}
				}
			}
		}
	}
}

resource "kubernetes_service" "my_app_service" {
	metadata {
		name = "my-app-service"
	}

	spec {
		type = "ClusterIP"
		selector = {
			app = "my-app"
		}

		port {
			protocol = "TCP"
			port = 80
			target_port = 80
		}
	}
}

resource "kubernetes_horizontal_pod_autoscaler" "my_app_hpa" {
	metadata {
		name = "my-app-hpa"
	}

	spec {
		min_replicas = 1
		max_replicas = 5

		scale_target_ref {
			api_version = "apps/v1"
			kind = "Deployment"
			name = "my-app-deployment"
		}

		metric {
			type = "Resource"

			resource {
				name = "cpu"

				target {
					type = "Utilization"
					average_utilization = 60
				}
			}
		}

		metric {
			type = "Resource"

			resource {
				name = "memory"

				target {
					type = "Utilization"
					average_utilization = 60
				}
			}
		}

	}
}

Steps to Reproduce

terraform apply (type yes)
Run terraform apply again without changing config

Expected Behavior

No changes should be listed as the config is unchanged.

Actual Behavior

Even though config is unchanged the actions below are listed on every terraform apply. If i change the order of metrics in the above config the changes to list memory above cpu, this does not happen.

  # kubernetes_horizontal_pod_autoscaler.my_app_deployment will be updated in-place
  ~ resource "kubernetes_horizontal_pod_autoscaler" "my_app_hpa" {
        id = "default/myapp-hpa"


      ~ spec {
            # (3 unchanged attributes hidden)

          ~ metric {
                # (1 unchanged attribute hidden)

              ~ resource {
                  ~ name = "memory" -> "cpu"

                    # (1 unchanged block hidden)
                }
            }
          ~ metric {
                # (1 unchanged attribute hidden)

              ~ resource {
                  ~ name = "cpu" -> "memory"

                    # (1 unchanged block hidden)
                }
            }

            # (1 unchanged block hidden)
        }
        # (1 unchanged block hidden)
    }
Plan: 0 to add, 1 to change, 0 to destroy.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

The text was updated successfully, but these errors were encountered:

jrhouston · 2021-05-05T06:47:25Z

It looks like the metrics do come back from the API in a different sort order – so we're missing a diff suppress function here.

Workaround is to reverse the order in the Terraform config.

joe-a-t · 2021-06-07T23:18:40Z

@jrhouston it doesn't look like reversing the order in the Terraform config so that memory is listed first and cpu is listed second makes a difference for me. One possible difference is that I am calling the kubernetes_horizontal_pod_autoscaler through a module and have dynamic metric blocks instead of hard coded. Is there any timeline or priority for this since it's pretty noisy and the workaround does not appear to work for people using dynamic metric blocks?

joe-a-t · 2021-08-26T23:17:57Z

@jrhouston is there any update on this? We are still experiencing the noise. Is this something where we could open a PR ourselves and get it reviewed? Is it literally just changing https://github.com/hashicorp/terraform-provider-kubernetes/blob/main/kubernetes/resource_kubernetes_horizontal_pod_autoscaler.go#L40 from TypeList to TypeSet? I can't tell if community PRs are accepted or not since there are currently 50 open PRs

cohandv · 2021-08-27T13:40:37Z

I also noticed there is no way to put the api_version that suppports multiple metrics

github-actions · 2022-08-28T00:00:41Z

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

joe-a-t · 2022-08-29T14:46:59Z

This issue is still a big problem, please address remove the stale label and address @jrhouston

…-kubernetes#1188

…der_to_hotfix_issue TD-590 Reverse order to temp fix issue hashicorp/terraform-provider-kubernetes#1188

github-actions · 2023-08-30T00:00:29Z

Marking this issue as stale due to inactivity. If this issue receives no comments in the next 30 days it will automatically be closed. If this issue was automatically closed and you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. This helps our maintainers find and focus on the active issues. Maintainers may also remove the stale label at their discretion. Thank you!

joe-a-t · 2023-08-30T16:09:59Z

This issue is still a big problem, please address remove the stale label and address @jrhouston

BBBmau · 2023-09-14T17:21:55Z

This is currently Blocked after discussing this issue further with @alexsomesan.

I attempted to solve this issue by switching to using Set instead of Lists, however this introduced another problem where an empty metric value gets added when attempting to Patch.

We discovered that the extra empty metric value comes from d.Get("metrics").(*Schema.Set) which comes from the terraform-plugin-sdk repo. Several issues are open that sound similar to the one being seen here, we'll come back to this issue once it has been addressed there.

hashicorp/terraform-plugin-sdk#588
hashicorp/terraform-plugin-sdk#1197

marjorg added the bug label Mar 8, 2021

dak1n1 added the help wanted label Mar 10, 2021

jrhouston mentioned this issue May 5, 2021

kubernetes_horizontal_pod_autoscaler: scaling based on cpu and memory produces updates each run hashicorp/terraform#28564

Closed

github-actions bot added the stale label Aug 28, 2022

github-actions bot removed the stale label Aug 29, 2022

arybolovlev mentioned this issue Mar 15, 2023

Multiple metrics on kubernetes_horizontal_pod_autoscaler_v2beta2 creates a permadiff #2039

Closed

tom-reinders added a commit to recognizegroup/terraform that referenced this issue May 8, 2023

TD-590 Reverse order to try hotfix issue hashicorp/terraform-provider…

383bdda

…-kubernetes#1188

tom-reinders mentioned this issue May 8, 2023

TD-590 Reverse order to temp fix issue hashicorp/terraform-provider-kubernetes#1188 recognizegroup/terraform#321

Merged

tom-reinders added a commit to recognizegroup/terraform that referenced this issue May 9, 2023

Merge pull request #321 from recognizegroup/feature/TD-590_reverse_or…

e422e04

…der_to_hotfix_issue TD-590 Reverse order to temp fix issue hashicorp/terraform-provider-kubernetes#1188

github-actions bot added the stale label Aug 30, 2023

github-actions bot removed the stale label Aug 30, 2023

BBBmau linked a pull request Sep 13, 2023 that will close this issue

Fix metrics hpa diffbug #2286

Open

2 tasks

BBBmau added upstream-terraform-sdk acknowledged Issue has undergone initial review and is in our work queue. labels Sep 14, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is it intentional that order of metrics matter on HPAs? #1188

Is it intentional that order of metrics matter on HPAs? #1188

marjorg commented Mar 8, 2021

jrhouston commented May 5, 2021

joe-a-t commented Jun 7, 2021 •

edited

joe-a-t commented Aug 26, 2021 •

edited

cohandv commented Aug 27, 2021

github-actions bot commented Aug 28, 2022

joe-a-t commented Aug 29, 2022

github-actions bot commented Aug 30, 2023

joe-a-t commented Aug 30, 2023

BBBmau commented Sep 14, 2023

Is it intentional that order of metrics matter on HPAs? #1188

Is it intentional that order of metrics matter on HPAs? #1188

Comments

marjorg commented Mar 8, 2021

Terraform Version, Provider Version and Kubernetes Version

Affected Resource(s)

Terraform Configuration Files

Steps to Reproduce

Expected Behavior

Actual Behavior

Community Note

jrhouston commented May 5, 2021

joe-a-t commented Jun 7, 2021 • edited

joe-a-t commented Aug 26, 2021 • edited

cohandv commented Aug 27, 2021

github-actions bot commented Aug 28, 2022

joe-a-t commented Aug 29, 2022

github-actions bot commented Aug 30, 2023

joe-a-t commented Aug 30, 2023

BBBmau commented Sep 14, 2023

joe-a-t commented Jun 7, 2021 •

edited

joe-a-t commented Aug 26, 2021 •

edited