Helm not detecting out-of-band changes/drift and reverting them upon apply

[rcng6514]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: 0.14.10 (can't yet upgrade - ongoing)
Provider version: 2.12.0
Kubernetes version: 1.24

Affected Resource(s)

• helm_release

Terraform Configuration Files

provider "helm" {
  kubernetes {
    host                   = local.host
    token                  = var.access_token == "" ? data.google_client_config.current.access_token : var.access_token
    cluster_ca_certificate = local.cluster_ca_certificate
  }
  experiments {
    manifest = true
  }
}

resource "helm_release" "flux" {
  name       = "flux"
  chart      = "${path.module}/charts/flux"
  namespace  = local.nwm_flux_namespace
  version    = "0.0.2"
  depends_on = [kubernetes_namespace.default, helm_release.flux_crds]

  set {
    name  = "git_path"
    value = local.git_path
  }

  set {
    name  = "iac_git_branch"
    value = var.iac_git_branch
  }

  set {
    name  = "iac_git_tag"
    value = var.iac_git_tag
  }

  set {
    name  = "network_policy_git_branch"
    value = var.network_policy_git_branch
  }

  set {
    name  = "network_policy_git_tag"
    value = var.network_policy_git_tag
  }

  set {
    name  = "git_semver"
    value = var.flux_semver
  }

  set {
    name  = "git_suspend"
    value = false
  }
}

Debug Output

�[0m�[1mmodule.default.module.default.helm_release.flux: Refreshing state... [id=flux]�[0m
2023-11-28T15:24:41.690Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceReleaseExists: flux] Start: timestamp=2023-11-28T15:24:41.690Z
2023-11-28T15:24:41.690Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [INFO] GetHelmConfiguration start: timestamp=2023-11-28T15:24:41.690Z
2023-11-28T15:24:41.691Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [INFO] Successfully initialized kubernetes config: timestamp=2023-11-28T15:24:41.691Z
2023-11-28T15:24:41.691Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [INFO] GetHelmConfiguration success: timestamp=2023-11-28T15:24:41.691Z
2023-11-28T15:24:41.691Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease wait for lock: timestamp=2023-11-28T15:24:41.691Z
2023-11-28T15:24:41.691Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease got lock, started: timestamp=2023-11-28T15:24:41.691Z
2023-11-28T15:24:41.691Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease post action created: timestamp=2023-11-28T15:24:41.691Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease post run: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease done: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceReleaseExists: flux] Done: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceReleaseRead: flux] Started: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [INFO] GetHelmConfiguration start: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [INFO] Successfully initialized kubernetes config: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [INFO] GetHelmConfiguration success: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease wait for lock: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease got lock, started: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.758Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease post action created: timestamp=2023-11-28T15:24:41.758Z
2023-11-28T15:24:41.817Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease post run: timestamp=2023-11-28T15:24:41.817Z
2023-11-28T15:24:41.817Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease done: timestamp=2023-11-28T15:24:41.817Z
2023-11-28T15:24:41.818Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceReleaseRead: flux] Done: timestamp=2023-11-28T15:24:41.818Z
2023-11-28T15:24:41.827Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceDiff: flux] Start: timestamp=2023-11-28T15:24:41.827Z
2023-11-28T15:24:41.827Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [INFO] GetHelmConfiguration start: timestamp=2023-11-28T15:24:41.827Z
2023-11-28T15:24:41.828Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [INFO] Successfully initialized kubernetes config: timestamp=2023-11-28T15:24:41.828Z
2023-11-28T15:24:41.828Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [INFO] GetHelmConfiguration success: timestamp=2023-11-28T15:24:41.828Z
2023-11-28T15:24:41.829Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceDiff: flux] Got chart: timestamp=2023-11-28T15:24:41.829Z
2023-11-28T15:24:41.829Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] Chart dependencies are up to date.: timestamp=2023-11-28T15:24:41.829Z
2023-11-28T15:24:41.829Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceDiff: flux] Release validated: timestamp=2023-11-28T15:24:41.829Z
2023-11-28T15:24:41.829Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease wait for lock: timestamp=2023-11-28T15:24:41.829Z
2023-11-28T15:24:41.829Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease got lock, started: timestamp=2023-11-28T15:24:41.829Z
2023-11-28T15:24:41.829Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease post action created: timestamp=2023-11-28T15:24:41.829Z
2023-11-28T15:24:41.882Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease post run: timestamp=2023-11-28T15:24:41.882Z
2023-11-28T15:24:41.882Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] flux getRelease done: timestamp=2023-11-28T15:24:41.882Z
2023-11-28T15:24:41.883Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 ---[ values.yaml ]-----------------------------------
git_path: prv2/gke
git_semver: ""
git_suspend: false
iac_git_branch: ""
iac_git_tag: master
network_policy_git_branch: ""
network_policy_git_tag: master: timestamp=2023-11-28T15:24:41.883Z
2023-11-28T15:24:41.883Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] [resourceDiff: flux] performing dry run upgrade: timestamp=2023-11-28T15:24:41.883Z
2023-11-28T15:24:41.886Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:41 [DEBUG] preparing upgrade for flux: timestamp=2023-11-28T15:24:41.886Z
2023-11-28T15:24:43.387Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:43 [DEBUG] performing update for flux: timestamp=2023-11-28T15:24:43.387Z
2023-11-28T15:24:44.038Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:44 [DEBUG] dry run for flux: timestamp=2023-11-28T15:24:44.037Z
s","path":"prv2/gke","prune":true,"serviceAccountName":"flux","sourceRef":{"kind":"GitRepository","name":"gke-cluster-services-network-policy","namespace":"nwm-flux-system"},"suspend":false,"validation":"none"}}}: timestamp=2023-11-28T15:24:44.038Z
2023-11-28T15:24:44.038Z [INFO]  plugin.terraform-provider-helm_v2.12.0_x5: 2023/11/28 15:24:44 [DEBUG] [resourceDiff: flux] Done: timestamp=2023-11-28T15:24:44.038Z
2023-11-28T15:24:44.042Z [WARN]  plugin.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2023-11-28T15:24:44.064Z [DEBUG] plugin: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/helm/2.12.0/linux_amd64/terraform-provider-helm_v2.12.0_x5 pid=495
2023-11-28T15:24:44.064Z [DEBUG] plugin: plugin exited.

Panic Output

N/A

Steps to Reproduce

1. terraform apply
2. Suspend Flux GitRepository CR via kubectl

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  annotations:
    meta.helm.sh/release-name: flux
    meta.helm.sh/release-namespace: nwm-flux-system
  creationTimestamp: "2022-03-28T17:49:24Z"
  finalizers:
  - finalizers.fluxcd.io
  generation: 128
  labels:
    app.kubernetes.io/managed-by: Helm
  name: gke-cluster-services
  namespace: nwm-flux-system
spec:
  interval: 1m0s
  ref:
    tag: master
  secretRef:
    name: iac-flux-secret
  suspend: true <<<
  timeout: 20s
  url: https://somewhere.only/we/know.git

3. Run apply

Expected Behavior

Helm should detect drift in GitRepository CR and revert suspend: true to suspend: false

Actual Behavior

No drift detected and no change triggered in apply to revert suspend to false

Important Factoids

• Only successful but undesirable workaround is to force a deploy each time:

  force_update = true
  set {
    name  = "update_timestamp"
    value = timestamp()
  }

References

• Add support for storing the rendered manifest (i.e Helm diff support) #702 - experimental feature enabled but not successful in detecting out-of-band changes

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment