set_list produces different behavior than array defined in values.yaml

[sourcehawk]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version:  1.6.2
Provider version: "~> 2"
Kubernetes version: 1.27

Affected Resource(s)

• helm_release (set_list)

Terraform Configuration Files

Relevant section displayed (two cases of set_list) with an empty array, which is also the default value within the chart itself, meaning it should have no effect

resource "helm_release" "monitoring" {
  chart = "./${path.module}/charts/monitoring"
  name = "monitoring"
  namespace = local.monitoring_namespace
  create_namespace = true
  timeout = 900 # 15 minutes

  set_list {
    name = "thanos.query.extraFlags"
    value = []
  }

  set_list {
    name = "thanos.query.stores"
    value = []
  }
}

Values.yaml file

thanos:
  query:
    enabled: true
    # Override default args
    extraFlags:
      []
      # Make client use TLS (internal now only works also with TLS)
      #- --grpc-client-tls-secure
      #- --endpoint=thanos-grpc.<this-cluster>.<domain>:443
      #- --endpoint=thanos-storegateway.<this-cluster>.<domain>:443
      #- --endpoint=thanos-ruler.<this-cluster>.<domain>:443
    # External store api's
    stores:
      []
      #- thanos-query.<external-cluster-1>.<domain>:443
      #- thanos-query.<external-cluster-2>.<domain>:443

Debug Output

Here in the "Args" section we can see that there is a newline before the "State" section. This newline is preventing the container from starting, as it errors upon startup because of an invalid start command.

Containers:
  query:
    Container ID:  containerd://26648b57a91e20f8f773a4d2953ab47fc70bb1b5c862d012b79599902bd2a642
    Image:         docker.io/bitnami/thanos:0.31.0-scratch-r8
    Image ID:      docker.io/bitnami/thanos@sha256:217e2d58f8b28ccf48fe75c54892c3090e6231eefd2094250d7e2012e6a0133b
    Ports:         10902/TCP, 10901/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      query
      --log.level=debug
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --query.replica-label=replica
      --endpoint=dnssrv+_grpc._tcp.monitoring-thanos-storegateway.monitoring.svc.cluster.local
      
    State:          Waiting
      Reason:       CrashLoopBackOff

Now if I comment out the set_list fields from earlier, this newline is no longer present. Even though an empty array is also the default value in my values.yaml file.

resource "helm_release" "monitoring" {
  chart = "./${path.module}/charts/monitoring"
  name = "monitoring"
  namespace = local.monitoring_namespace
  create_namespace = true
  timeout = 900 # 15 minutes

  #set_list {
  #  name = "thanos.query.extraFlags"
  #  value = []
  #}

  #set_list {
  #  name = "thanos.query.stores"
  #  value = [ ]
  #}
}

Containers:
  query:
    Container ID:  containerd://ea082134315feb72d48150d6db3ac07da5cdf7d6c141c61a88c826e2ae4f8500
    Image:         docker.io/bitnami/thanos:0.31.0-scratch-r8
    Image ID:      docker.io/bitnami/thanos@sha256:217e2d58f8b28ccf48fe75c54892c3090e6231eefd2094250d7e2012e6a0133b
    Ports:         10902/TCP, 10901/TCP
    Host Ports:    0/TCP, 0/TCP
    Args:
      query
      --log.level=debug
      --log.format=logfmt
      --grpc-address=0.0.0.0:10901
      --http-address=0.0.0.0:10902
      --query.replica-label=replica
      --endpoint=dnssrv+_grpc._tcp.monitoring-thanos-storegateway.monitoring.svc.cluster.local
    State:          Running
      Started:      Wed, 18 Oct 2023 22:02:33 +0000

Update:

If I go into edit mode using kubectl edit on the pod, there seems to be an additional array element added. Again, this does not happen when deploying using only values.yaml file with the same values, so the behavior in set_list is clearly not the same.

  containers:
  - args:
    - query
    - --log.level=info
    - --log.format=logfmt
    - --grpc-address=0.0.0.0:10901
    - --http-address=0.0.0.0:10902
    - --query.replica-label=replica
    - --endpoint=monitoring-thanos-discovery:10901
    - --endpoint=monitoring-thanos-storegateway:10901
    - --endpoint=monitoring-thanos-receive:10901
    - --endpoint=thanos-ruler-operated:10901
    - ""

Expected Behavior

The set_list only creates lists and does not add anything that is not expected.

Actual Behavior

The set_list creates an unwanted newline where it is not expected.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[arybolovlev]

Hi @hauks96,

This provider uses Helm ParseInto function to parse set_list values. If the list is empty, it produces an empty string that matches Helm CLI behavior. The new lines you observe most probably come from the chart template. They might not handle this case scenario well and don't trim a new line. This is something that the provider cannot do and is totally up to the chart developers.

Have you tried to run helm template for the same chart and pass empty lists for the mentioned values explicitly? Please make sure you use the same chart version.

P.S.: I have tried the following config and don't see any issues.

resource "helm_release" "this" {
  name       = "this"
  repository = "oci://registry-1.docker.io"
  chart      = "bitnamicharts/thanos"
  namespace  = "default"
  timeout    = 900

  set_list {
    name  = "thanos.query.extraFlags"
    value = []
  }

  set_list {
    name  = "thanos.query.stores"
    value = []
  }
}

$ helm list
NAME	NAMESPACE	REVISION	UPDATED                              	STATUS  	CHART          	APP VERSION
this	default  	1       	2023-10-23 14:08:43.442646 +0200 CEST	deployed	thanos-12.13.12	0.32.5

Here is the chart I used to reproduce: https://artifacthub.io/packages/helm/bitnami/thanos

[sourcehawk]

ages/helm/bitnami/thanos

Try it without the "thanos" path. I am using "thanos" in my path because I am using it as a subchart. Your assignment here will not have any effect because you are not setting the right attributes.

resource "helm_release" "this" {
  name       = "this"
  repository = "oci://registry-1.docker.io"
  chart      = "bitnamicharts/thanos"
  namespace  = "default"
  timeout    = 900

  set_list {
    name  = "query.extraFlags"
    value = []
  }

  set_list {
    name  = "query.stores"
    value = []
  }
}

[arybolovlev]

Hi @hauks96,

Thank you for the correction. However, my answer remains the same. An empty list produces an empty line(this matches with Helm CLI) and the observed behavior is how chart developers handle this case.

Thanks.

[sourcehawk]

My problem here is the fact that using a values.yaml file produces a different output from set_list with the exact same value. I don't quite understand how that is the chart templates fault.

[sourcehawk]

@arybolovlev Can you please re-add the bug label? This is clearly not intended behavior, given that it is not matching the behavior of vanilla helm values file. I've resorted to using the values section of helm_release to set the arrays in this case to prevent the issue from happening.