Plugin Crash with Error: The terraform-provider-kubernetes_v2.27.0_x5 plugin crashed

[GaneshKudikala]

The terraform-provider-kubernetes plugin crashed with the following error message:

Error: The terraform-provider-kubernetes_v2.27.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Terraform Version, Provider Version and Kubernetes Version

Terraform version: Terraform v1.6.4
Kubernetes provider version: kubernetes v2.27.0
Kubernetes version: [Client Version: v1.28.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.0-eks-c417bb3]

Affected Resource(s)

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.


data "kubernetes_resources" "deployments" {
   for_each    = toset(data.kubernetes_all_namespaces.allns.namespaces)
    api_version = "apps/v1"
  # Query all deployments
  kind = "Deployment"

  # Optionally, filter deployments by namespace
  namespace = each.key
}

provider "kubernetes" {
    config_path    = "~/.kube/config"
}

data "kubernetes_all_namespaces" "allns" {
}

Here is the Terraform File, The issue occors only if the Deployment/apps have 3 or more deployments, So if you want to recreate the error, Deploy EKS cluster with each namespace with 3+ depployments are may be you can see the error.

Debug Output

https://gist.github.com/GaneshKudikala/3a97545ec3cb4776c55754a4d957314d

Panic Output

Steps to Reproduce

Expected Behavior

What should have happened?

Actual Behavior

What actually happened?

Important Factoids

References

• Missing secretNamespace on azure_file of persistent_volume_source #1234
  Include conditions in nodes data source #2430

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[BBBmau]

@GaneshKudikala Hi thanks for opening this issue. Can you provide us with the YAML that you've used for your deployments? It will help with reproducing this.

[GaneshKudikala]

To resproduce the same error I have made the following steps:

1. Deploy a Kind Kuberntes cluster with following configuration

# three node (two workers) cluster config
  kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
  - role: control-plane
  - role: worker
  - role: worker

3. Deploy sample applications from the repo https://github.com/digitalocean/kubernetes-sample-apps
   kubectl apply -k bookinfo-example/kustomize
kubectl apply -k emojivoto-example/kustomize
kubectl apply -k podinfo-example/kustomize

4. The following terraform code was used to retrive the resources from the cluster for each namespaces.
   `

data "kubernetes_resources" "deployments" {
  for_each = toset(data.kubernetes_all_namespaces.allns.namespaces)
  api_version = "apps/v1"
  kind        = "Deployment"
  namespace   = each.key
}

provider "kubernetes" {
 config_path    = "~/.kube/config"
  config_context = "kind-k8s-playground"

}


data "kubernetes_all_namespaces" "allns" {}`

5. Try terraform init and terraform plan to give us the Error

`goroutine 67 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
	github.com/hashicorp/terraform-plugin-go@v0.20.0/tftypes/value.go:278
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.UnknownToNull({{0x24c8ec0?, 0xc00a27f1d0?}, {0x1e62300?, 0xc00a25a5d0?}})
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:143 +0x508
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.UnknownToNull({{0x24c8ec0?, 0xc00a305c50?}, {0x1e62300?, 0xc00a25a540?}})
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:141 +0x438
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.UnknownToNull({{0x24c8ec0?, 0xc00a30b710?}, {0x1e62300?, 0xc00a259710?}})
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:141 +0x438
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.UnknownToNull({{0x24c8ec0?, 0xc00a3120c0?}, {0x1e62300?, 0xc00a243770?}})
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:141 +0x438
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.UnknownToNull({{0x24c9128?, 0xc007b5f020?}, {0x1d393a0?, 0xc00470a1f8?}})
	github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:134 +0x566
github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).ReadPluralDataSource(0xc00019ca80, {0x24c1f48, 0xc000f21f20}, 0xc0004b9480)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/datasource.go:218 +0x1c11
github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).ReadDataSource(0x24c1f48?, {0x24c1f48, 0xc000f21f20}, 0xc0004b9480)
	github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/datasource.go:28 +0x85
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.(*muxServer).ReadDataSource(0x24c1f80?, {0x24c1f48?, 0xc000f21c20?}, 0xc0004b9480)
	github.com/hashicorp/terraform-plugin-mux@v0.12.0/tf5muxserver/mux_server_ReadDataSource.go:36 +0x19f
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadDataSource(0xc000468500, {0x24c1f48?, 0xc000f21470?}, 0xc0004d2190)
	github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/tf5server/server.go:699 +0x3df
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadDataSource_Handler({0x20c32c0?, 0xc000468500}, {0x24c1f48, 0xc000f21470}, 0xc0002b5380, 0x0)
	github.com/hashicorp/terraform-plugin-go@v0.20.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:539 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000b7e000, {0x24c1f48, 0xc000f213e0}, {0x24ca2a0, 0xc00074b6c0}, 0xc000f27e60, 0xc000b822a0, 0x3546dc8, 0x0)
	google.golang.org/grpc@v1.60.0/server.go:1372 +0xe03
google.golang.org/grpc.(*Server).handleStream(0xc000b7e000, {0x24ca2a0, 0xc00074b6c0}, 0xc000f27e60)
	google.golang.org/grpc@v1.60.0/server.go:1783 +0xfec
google.golang.org/grpc.(*Server).serveStreams.func2.1()
	google.golang.org/grpc@v1.60.0/server.go:1016 +0x59
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 25
	google.golang.org/grpc@v1.60.0/server.go:1027 +0x115

Error: The terraform-provider-kubernetes_v2.27.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.
`

[xego]

This issue is not version specific, we had the same behavior with previous versions of the kubernetes provider.

[PavelSlepushkin]

It looks like manifects/data sources - kubernetes_resources are broken. it failed for me with latest version of provider (v.2.29.0) with example from documentation (getting all namespaces via manifest/kubernetes_resources datasource):
https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/data-sources/resources#example-get-a-list-of-namespaces-excluding-kube-system-using-field_selector
I aslo can confim from TF_TRACE that data was fetched from k8s and it fails in
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.UnknownToNull
I guess it's the same issue and there's no reason to create another one.

Also, this can be related/same for #2215

[alexsomesan]

Thank you for the extensive instructions, @GaneshKudikala
I was able to reproduce following your steps.
It seems we already had an open PR that addresses this issue and it somehow slipped through the cracks and never got merged.

The fix for this will be available in the next provider release.

To resproduce the same error I have made the following steps:

...

[GaneshKudikala]

@alexsomesan I have seen the new release version v2.30.0, i have upgraded my plugin to use the new version however this new version may be doesn't have the fix(or Correct me if i am wrong), Still crashing