ignore_changes not working in kubernetes_manifest

[thecosmicfrog]

Terraform version, Kubernetes provider version and Kubernetes version

Terraform version: Terraform v1.5.5 on darwin_arm64
Kubernetes Provider version: v2.25.2
Kubernetes version: 1.28

Terraform configuration

locals {
  applications = {
    application-1 = {
      chart = "application-1"
    }
    application-2 = {
      chart = "application-2"
    }
    application-3 = {
      chart   = "application-3"
      version = 3.46.0
    }
  }
}

resource "kubernetes_manifest" "argocd_application" {
  for_each = local.applications

  manifest = {
    apiVersion = "argoproj.io/v1alpha1"
    kind       = "Application"

    metadata = {
      name      = each.key
      namespace = "kube-system"

      finalizers = [
        # Foreground cascading deletion.
        "resources-finalizer.argocd.argoproj.io"
      ]
    }

    spec = {
      project = "default"

      source = {
        repoURL        = "https://artifactory.example.com/artifactory/helm-all"
        chart          = each.value["chart"]
        targetRevision = try(var.charts[each.value["chart"]]["version"], null)

        helm = {
          valuesObject = {
            replicas = 1
          }

          parameters = concat([
            {
              name = "image.name"
              value = each.key
            }
          ],
            try(each.value["version"], null) != null ? [
              {
                name = "image.tag"
                value = each.value["version"]
              }
            ] : [],
          )
        }
      }

      destination = {
        server    = "https://kubernetes.default.svc"
        namespace = "mynamespace"
      }

      syncPolicy = {
        automated = {
          prune    = true
          selfHeal = true
        }
      }
    }
  }
}

Question

Hi all. I am using the kubernetes_manifest resource in order to install Argo CD Application objects (spec is defined here).

Initial object creation works without issue, and subsequent runs of terraform apply show no changes (as I would expect).

However, if the input value of manifest.spec.source.helm.valuesObject changes (say, the value of foo changes from bar to bar123), the subsequent run of terraform apply says that the entire kubernetes_manifest resource must be replaced, alongside a massive number of added and changed fields:

Terraform will perform the following actions:

  # kubernetes_manifest.argocd_application["application-3"] must be replaced
-/+ resource "kubernetes_manifest" "argocd_application" {
      ~ manifest = {
          ~ spec       = {
              ~ source      = {
                  ~ helm           = {
                      ~ valuesObject = {
                          ~ foo                   = "bar" -> "bar123"
                            # (6 unchanged attributes hidden)
                        }
                        # (1 unchanged attribute hidden)
                    }
                    # (3 unchanged attributes hidden)
                }
                # (3 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
      ~ object   = {
          ~ metadata   = {
              + annotations                = (known after apply)
              + creationTimestamp          = (known after apply)
              + deletionGracePeriodSeconds = (known after apply)
              + deletionTimestamp          = (known after apply)
              + generateName               = (known after apply)
              + generation                 = (known after apply)
              + labels                     = (known after apply)
              + managedFields              = (known after apply)
                name                       = "application-3"
              + ownerReferences            = (known after apply)
              + resourceVersion            = (known after apply)
              + selfLink                   = (known after apply)
              + uid                        = (known after apply)
                # (2 unchanged attributes hidden)
            }
          ~ operation  = {
              + info        = (known after apply)
              ~ initiatedBy = {
                  + automated = (known after apply)
                  + username  = (known after apply)
                }
              ~ retry       = {
                  ~ backoff = {
                      + duration    = (known after apply)
                      + factor      = (known after apply)
                      + maxDuration = (known after apply)
                    }
                  + limit   = (known after apply)
                }
              ~ sync        = {
                  + dryRun       = (known after apply)
                  + manifests    = (known after apply)
                  + prune        = (known after apply)
                  + resources    = (known after apply)
                  + revision     = (known after apply)
                  + revisions    = (known after apply)
                  ~ source       = {
                      + chart          = (known after apply)
                      ~ directory      = {
                          + exclude = (known after apply)
                          + include = (known after apply)
                          ~ jsonnet = {
                              + extVars = (known after apply)
                              + libs    = (known after apply)
                              + tlas    = (known after apply)
                            }
                          + recurse = (known after apply)
                        }
                      ~ helm           = {
                          + fileParameters          = (known after apply)
                          + ignoreMissingValueFiles = (known after apply)
                          + parameters              = (known after apply)
                          + passCredentials         = (known after apply)
                          + releaseName             = (known after apply)
                          + skipCrds                = (known after apply)
                          + valueFiles              = (known after apply)
                          + values                  = (known after apply)
                          + valuesObject            = (known after apply)
                          + version                 = (known after apply)
                        }
                      ~ kustomize      = {
                          + commonAnnotations         = (known after apply)
                          + commonAnnotationsEnvsubst = (known after apply)
                          + commonLabels              = (known after apply)
                          + forceCommonAnnotations    = (known after apply)
                          + forceCommonLabels         = (known after apply)
                          + images                    = (known after apply)
                          + namePrefix                = (known after apply)
                          + nameSuffix                = (known after apply)
                          + namespace                 = (known after apply)
                          + patches                   = (known after apply)
                          + replicas                  = (known after apply)
                          + version                   = (known after apply)
                        }
                      + path           = (known after apply)
                      ~ plugin         = {
                          + env        = (known after apply)
                          + name       = (known after apply)
                          + parameters = (known after apply)
                        }
                      + ref            = (known after apply)
                      + repoURL        = (known after apply)
                      + targetRevision = (known after apply)
                    }
                  + sources      = (known after apply)
                  + syncOptions  = (known after apply)
                  ~ syncStrategy = {
                      ~ apply = {
                          + force = (known after apply)
                        }
                      ~ hook  = {
                          + force = (known after apply)
                        }
                    }
                }
            }
          ~ spec       = {
              ~ destination          = {
                  + name      = (known after apply)
                    # (2 unchanged attributes hidden)
                }
              + ignoreDifferences    = (known after apply)
              + info                 = (known after apply)
              + revisionHistoryLimit = (known after apply)
              ~ source               = {
                  ~ directory      = {
                      + exclude = (known after apply)
                      + include = (known after apply)
                      ~ jsonnet = {
                          + extVars = (known after apply)
                          + libs    = (known after apply)
                          + tlas    = (known after apply)
                        }
                      + recurse = (known after apply)
                    }
                  ~ helm           = {
                      + fileParameters          = (known after apply)
                      + ignoreMissingValueFiles = (known after apply)
                      ~ parameters              = [
                          ~ {
                              + forceString = (known after apply)
                                name        = "image.name"
                                # (1 unchanged attribute hidden)
                            },
                          ~ {
                              + forceString = (known after apply)
                                name        = "image.tag"
                                # (1 unchanged attribute hidden)
                            },
                        ]
                      + passCredentials         = (known after apply)
                      + releaseName             = (known after apply)
                      + skipCrds                = (known after apply)
                      + valueFiles              = (known after apply)
                      + values                  = (known after apply)
                      ~ valuesObject            = {
                          + foo                            = "bar"
                            # (7 unchanged attributes hidden)
                        }
                      + version                 = (known after apply)
                    }
                  ~ kustomize      = {
                      + commonAnnotations         = (known after apply)
                      + commonAnnotationsEnvsubst = (known after apply)
                      + commonLabels              = (known after apply)
                      + forceCommonAnnotations    = (known after apply)
                      + forceCommonLabels         = (known after apply)
                      + images                    = (known after apply)
                      + namePrefix                = (known after apply)
                      + nameSuffix                = (known after apply)
                      + namespace                 = (known after apply)
                      + patches                   = (known after apply)
                      + replicas                  = (known after apply)
                      + version                   = (known after apply)
                    }
                  + path           = (known after apply)
                  ~ plugin         = {
                      + env        = (known after apply)
                      + name       = (known after apply)
                      + parameters = (known after apply)
                    }
                  + ref            = (known after apply)
                    # (3 unchanged attributes hidden)
                }
              + sources              = (known after apply)
              ~ syncPolicy           = {
                  ~ automated                = {
                      + allowEmpty = (known after apply)
                        # (2 unchanged attributes hidden)
                    }
                  ~ managedNamespaceMetadata = {
                      + annotations = (known after apply)
                      + labels      = (known after apply)
                    }
                  ~ retry                    = {
                      ~ backoff = {
                          + duration    = (known after apply)
                          + factor      = (known after apply)
                          + maxDuration = (known after apply)
                        }
                      + limit   = (known after apply)
                    }
                  + syncOptions              = (known after apply)
                }
                # (1 unchanged attribute hidden)
            }
            # (2 unchanged attributes hidden)
        }
    }

I understand that objects created by Terraform can change outside of its control, and it appears that Argo CD's controllers are doing that in this case, by creating default values for optional fields. As such, I added a few lifecycle.ignore_changes entries to instruct Terraform to disregard these:

resource "kubernetes_manifest" "argocd_application" {
  lifecycle {
    ignore_changes = [
      object.metadata.annotations,
      object.metadata.finalizers,
      object.metadata.labels,
      object.operation.initiatedBy.automated
    ]
  }

  [... rest of resource...]
}

However, this appears to have made no difference, as a terraform apply still gives me the same "must be replaced" output with all the same fields.

I feel like I may be missing something here, so I'm asking for assistance at this stage. Thanks! 🙂

[jrhouston]

Hi @thecosmicfrog – terraform's ignore_changes feature doesn't work with fields nested within the object attribute. This provider implements it's own computed_fields attribute which you could use to specify the fields which are being mutated by the ArgoCD controller, see here: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/manifest#computed-fields

[thecosmicfrog]

Hi @jrhouston. Thanks for your response.

I have experimented with several computed_fields inputs and am still seeing the "must be replaced" behaviour. The only difference I see when running terraform apply is that the non-Terraform-specified fields are to be deleted instead of added (note the - symbols instead of + symbols) in output below.

New Terraform config and output:

Terraform configuration

resource "kubernetes_manifest" "argocd_application" {
  for_each = local.applications

  # Trying multiple fields.
  computed_fields = [
    "object.metadata",
    "object.operation",
    "object.spec",
    "metadata",
    "operation",
    "spec"
  ]

  <removed lifecycle.ignore_changes block>

  [...rest of resource as in original question...]
}

terraform apply output

Terraform will perform the following actions:

  # kubernetes_manifest.argocd_application["application-3"] must be replaced
-/+ resource "kubernetes_manifest" "argocd_application" {
      ~ manifest        = {
          ~ spec       = {
              ~ source      = {
                  ~ helm           = {
                      ~ valuesObject = {
                          ~ foo = bar -> bar123
                        }
                        # (1 unchanged attribute hidden)
                    }
                    # (3 unchanged attributes hidden)
                }
                # (3 unchanged attributes hidden)
            }
            # (3 unchanged attributes hidden)
        }
      ~ object          = {
          ~ metadata   = {
              - annotations                = null
              - creationTimestamp          = null
              - deletionGracePeriodSeconds = null
              - deletionTimestamp          = null
              - finalizers                 = [
                  - "resources-finalizer.argocd.argoproj.io",
                ]
              - generateName               = null
              - generation                 = null
              - labels                     = null
              - managedFields              = null
              - name                       = "application-3"
              - namespace                  = "kube-system"
              - ownerReferences            = null
              - resourceVersion            = null
              - selfLink                   = null
              - uid                        = null
            } -> (known after apply)
          ~ operation  = {
              - info        = null
              - initiatedBy = {
                  - automated = null
                  - username  = null
                }
              - retry       = {
                  - backoff = {
                      - duration    = null
                      - factor      = null
                      - maxDuration = null
                    }
                  - limit   = null
                }
              - sync        = {
                  - dryRun       = null
                  - manifests    = null
                  - prune        = null
                  - resources    = null
                  - revision     = null
                  - revisions    = null
                  - source       = {
                      - chart          = null
                      - directory      = {
                          - exclude = null
                          - include = null
                          - jsonnet = {
                              - extVars = null
                              - libs    = null
                              - tlas    = null
                            }
                          - recurse = null
                        }
                      - helm           = {
                          - fileParameters          = null
                          - ignoreMissingValueFiles = null
                          - parameters              = null
                          - passCredentials         = null
                          - releaseName             = null
                          - skipCrds                = null
                          - valueFiles              = null
                          - values                  = null
                          - valuesObject            = null
                          - version                 = null
                        }
                      - kustomize      = {
                          - commonAnnotations         = null
                          - commonAnnotationsEnvsubst = null
                          - commonLabels              = null
                          - forceCommonAnnotations    = null
                          - forceCommonLabels         = null
                          - images                    = null
                          - namePrefix                = null
                          - nameSuffix                = null
                          - namespace                 = null
                          - patches                   = null
                          - replicas                  = null
                          - version                   = null
                        }
                      - path           = null
                      - plugin         = {
                          - env        = null
                          - name       = null
                          - parameters = null
                        }
                      - ref            = null
                      - repoURL        = null
                      - targetRevision = null
                    }
                  - sources      = null
                  - syncOptions  = null
                  - syncStrategy = {
                      - apply = {
                          - force = null
                        }
                      - hook  = {
                          - force = null
                        }
                    }
                }
            } -> (known after apply)
          ~ spec       = {
              - destination          = {
                  - name      = null
                  - namespace = "mynamespace"
                  - server    = "https://kubernetes.default.svc"
                }
              - ignoreDifferences    = null
              - info                 = null
              - project              = "default"
              - revisionHistoryLimit = null
              - source               = {
                  - chart          = "standard-base-chart"
                  - directory      = {
                      - exclude = null
                      - include = null
                      - jsonnet = {
                          - extVars = null
                          - libs    = null
                          - tlas    = null
                        }
                      - recurse = null
                    }
                  - helm           = {
                      - fileParameters          = null
                      - ignoreMissingValueFiles = null
                      - parameters              = [
                          - {
                              - forceString = null
                              - name        = "image.name"
                              - value       = "application-3"
                            },
                          - {
                              - forceString = null
                              - name        = "image.tag"
                              - value       = "3.46.0"
                            },
                        ]
                      - passCredentials         = null
                      - releaseName             = null
                      - skipCrds                = null
                      - valueFiles              = null
                      - values                  = null
                      - valuesObject            = {
                          - foo = bar
                        }
                      - version                 = null
                    }
                  - kustomize      = {
                      - commonAnnotations         = null
                      - commonAnnotationsEnvsubst = null
                      - commonLabels              = null
                      - forceCommonAnnotations    = null
                      - forceCommonLabels         = null
                      - images                    = null
                      - namePrefix                = null
                      - nameSuffix                = null
                      - namespace                 = null
                      - patches                   = null
                      - replicas                  = null
                      - version                   = null
                    }
                  - path           = null
                  - plugin         = {
                      - env        = null
                      - name       = null
                      - parameters = null
                    }
                  - ref            = null
                  - repoURL        = "https://artifactory.example.com/artifactory/helm-all"
                  - targetRevision = "76.0.0"
                }
              - sources              = null
              - syncPolicy           = {
                  - automated                = {
                      - allowEmpty = null
                      - prune      = true
                      - selfHeal   = true
                    }
                  - managedNamespaceMetadata = {
                      - annotations = null
                      - labels      = null
                    }
                  - retry                    = {
                      - backoff = {
                          - duration    = null
                          - factor      = null
                          - maxDuration = null
                        }
                      - limit   = null
                    }
                  - syncOptions              = [
                      - "ServerSideApply=true",
                    ]
                }
            } -> (known after apply)
            # (2 unchanged attributes hidden)
        }
        # (1 unchanged attribute hidden)
    }

Plan: 1 to add, 0 to change, 1 to destroy.

As before, this "must be replaced" behaviour occurs every time I update the valuesObject.

Thanks,
Aaron

[BBBmau]

This issue is related to #2375, the valuesObject field is set to x-kubernetes-preserve-unknown-fields which causes a must replace behavior.

[thecosmicfrog]

@BBBmau Interesting, thanks for sharing! I've subscribed to the mentioned issue. Are there any suggested paths forward at this time?

[BBBmau]

@thecosmicfrog unfortunately as of right now there is no workaround since this is by design to prevent any potential crashes from occurring since x-kubernetes-preserve-unknown-fields can be seen as a wild card in manifest.

A redesign is in discussion and will be worked on to address this issue. Just a matter of patience!

[thecosmicfrog]

Thanks @BBBmau. Will this redesign be a feature of terraform-provider-kubernetes or Terraform itself?

[BBBmau]

@thecosmicfrog it will be a feature implemented into the terraform-provider-kubernetes

[thecosmicfrog]

@thecosmicfrog it will be a feature implemented into the terraform-provider-kubernetes

Great, thanks @BBBmau!