terraform destroy Invalid index issue #1161
Comments
armujahid
changed the title
|
I have the same problem ❯ terraform -chdir=tf-aws-eks-blueprint destroy -var-file=../cluster-k8s-sbx/variables.tfvars -target="module.eks_blueprints"
module.eks_blueprints.data.aws_partition.current: Reading...
module.eks_blueprints.module.aws_eks.module.kms.data.aws_caller_identity.current: Reading...
module.eks_blueprints.module.aws_eks.module.kms.data.aws_partition.current: Reading...
data.aws_subnets.private: Reading...
module.eks_blueprints.data.aws_region.current: Reading...
module.eks_blueprints.module.aws_eks.data.aws_partition.current: Reading...
data.aws_vpc.selected: Reading...
module.eks_blueprints.module.aws_eks.data.aws_caller_identity.current: Reading...
module.eks_blueprints.data.aws_caller_identity.current: Reading...
module.eks_blueprints.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints.module.aws_eks.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.eks_blueprints.module.aws_eks.module.kms.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints.module.aws_eks.data.aws_iam_policy_document.assume_role_policy[0]: Reading...
module.eks_blueprints.module.aws_eks.data.aws_iam_policy_document.assume_role_policy[0]: Read complete after 0s [id=XXXXXXXX]
data.aws_subnets.private: Read complete after 0s [id=eu-central-1]
data.aws_vpc.selected: Read complete after 0s [id=vpc-XXXXXXXX]
module.eks_blueprints.module.aws_eks.module.kms.data.aws_caller_identity.current: Read complete after 0s [id=XXXXXXXX]
module.eks_blueprints.data.aws_caller_identity.current: Read complete after 0s [id=XXXXXXXX]
module.eks_blueprints.module.aws_eks.data.aws_caller_identity.current: Read complete after 0s [id=XXXXXXXX]
module.eks_blueprints.data.aws_iam_session_context.current: Reading...
module.eks_blueprints.data.aws_iam_session_context.current: Read complete after 1s [id=arn:aws:sts::XXXXXXXX:assumed-role/XXXXXXXXXXXX]
module.eks_blueprints.data.aws_iam_policy_document.eks_key: Reading...
module.eks_blueprints.data.aws_iam_policy_document.eks_key: Read complete after 0s [id=XXXXXXXXXXXXX]
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵
╷
│ Error: Invalid index
│
│ on .terraform/modules/eks_blueprints.aws_eks/main.tf line 125, in locals:
│ 125: cluster_security_group_id = local.create_cluster_sg ? aws_security_group.cluster[0].id : var.cluster_security_group_id
│ ├────────────────
│ │ aws_security_group.cluster is empty tuple
│
│ The given key does not identify an element in this collection value: the collection has no elements.
╵
╷
│ Error: Invalid index
│
│ on .terraform/modules/eks_blueprints.aws_eks/node_groups.tf line 59, in locals:
│ 59: node_security_group_id = local.create_node_sg ? aws_security_group.node[0].id : var.node_security_group_id
│ ├────────────────
│ │ aws_security_group.node is empty tuple
│
│ The given key does not identify an element in this collection value: the collection has no elements.
╵After having the problem the first time, I destroyed TF resources manually until I reach this minimal set. The issue still persists: ❯ terraform -chdir=tf-aws-eks-blueprint state list
data.aws_ami.eks
data.aws_autoscaling_groups.eks_node_group
data.aws_availability_zones.available
data.aws_caller_identity.current
data.aws_eks_addon_version.default["kube-proxy"]
data.aws_eks_addon_version.latest["coredns"]
data.aws_eks_addon_version.latest["vpc-cni"]
data.aws_eks_cluster.cluster
data.aws_eks_cluster_auth.cluster
data.aws_iam_policy_document.argocd_reposerver_iam_policy_document
data.aws_iam_policy_document.aws_ebs_csi_driver_iam_policy_document
data.aws_iam_policy_document.aws_lb_iam_policy_document
data.aws_iam_policy_document.crossplane_iam_policy_document
data.aws_iam_policy_document.external_dns_iam_policy_document
data.aws_iam_policy_document.karpenter_controller_iam_policy_document
data.aws_partition.current
data.aws_region.current
data.aws_route53_zone.public-dns-zone
data.aws_security_groups.gitlab-forref
data.aws_subnets.private
data.aws_vpc.selected
data.kubectl_path_documents.crds_monitoring_coreos_com
aws_acm_certificate_validation.private-cluster-cert
aws_acm_certificate_validation.public-cluster-cert
aws_route53_record.private-cluster-cert["*.k8s.private.XXXXXXXXXXXX.com"]
aws_route53_record.public-cluster-cert["*.k8s.XXXXXXXXXXXX.com"]
module.eks_blueprints.data.aws_caller_identity.current
module.eks_blueprints.data.aws_partition.current
module.eks_blueprints.data.aws_region.currentProblem exists with module version: TF provider plugin versions: latest - Using previously-installed hashicorp/time v0.9.1
- Using previously-installed okta/okta v3.38.0
- Using previously-installed hashicorp/random v3.4.3
- Using previously-installed gavinbunney/kubectl v1.14.0
- Using previously-installed hashicorp/helm v2.7.1
- Using previously-installed hashicorp/cloudinit v2.2.0
- Using previously-installed terraform-aws-modules/http v2.4.1
- Using previously-installed hashicorp/kubernetes v2.15.0
- Using previously-installed hashicorp/aws v4.39.0
- Installing hashicorp/tls v4.0.4...
- Installed hashicorp/tls v4.0.4 (signed by HashiCorp)
- Using previously-installed hashicorp/local v2.2.3
- Using previously-installed hashicorp/null v3.2.0Edit 2022-11-16: After some more research it seems that this is rather a problem with the AWS EKS TF module: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/main.tf#L125 It looks related to this: terraform-aws-modules/terraform-aws-eks#568 The cluster is already deleted but still appears in the state of |
|
I started seeing this after a recent upgrade from 4.14.0 to 4.15.0 of terraform-aws-eks-blueprints. The error I'm seeing is: EDIT: I see now that my mention of cluster SG is probably irrelevant here; the issue is related to the node security group. In contrast with @armujahid's environment, I'm creating my own Security Group to pass to |
|
@ayeks @armujahid are you letting eks_blueprints EDIT: I see now that cluster SG is probably irrelevant here; the issue is related to the node security group. @ayeks what specifically are you trying to link to in aws_eks module code and that closed issue. |
|
@timblaktu I am letting eks_blueprint to create everything from scratch by passing nothing (since |
|
@ayeks I see in the changeset that resolved aws-eks#568 (same issue, but for cluster sg, not node sg) they basically changed references to the cluster list to use So, it would seem the same needs to be done for However, this implies that this problem is long-standing (since that issue is a couple years old), yet I only started seeing the issue recently after upgrading from 4.14.0 to 4.15.0 of terraform-aws-eks-blueprints. I can see that the 4.14.0-->4.15.0 upgrade includes an upgrade of the aws_eks module from 18.26.6-->18.29.1. Tracing through the terraform-aws-eks changelog, I see that there was a bugfix in the penultimate release 18.30.2, that addressed this issue, which seems relevant because the changeset touches node security groups and cluster security group rules. Since I'm already using a fork of eks-blueprints to work around other issues, I'm going to upgrade to terraform-aws-eks 18.30.2 as a simple test. |
|
Latest module version didn't fix it for me, so I created this fork/branch with the fix that I think is appropriate. Testing now. |
|
@timblaktu thanks for the detailed response and investigation! I let However, I pass the SG ID to module module "eks_blueprints_kubernetes_addons" {
source = "github.com/aws-ia/terraform-aws-eks-blueprints//modules/kubernetes-addons?ref=v4.16.0"
eks_cluster_id = module.eks_blueprints.eks_cluster_id
eks_worker_security_group_id = module.eks_blueprints.worker_node_security_group_id
..And I also store the SG ID in a AWS Secrets Manager Secret - but this object is already deleted in AWS and does not exist in TF state anymore. So I think its not relevant. resource "aws_secretsmanager_secret_version" "k8s-generic-cluster-secret" {
secret_id = aws_secretsmanager_secret.k8s-generic-cluster-secret.id
secret_string = jsonencode({
"karpenter-endpoint": data.aws_eks_cluster.cluster.endpoint,
"gitlab-runner-sg": "${data.aws_eks_cluster.cluster.vpc_config[0].cluster_security_group_id},${data.aws_security_groups.gitlab-forref.ids[0]}"
})
}I deleted everything and deployed the cluster again to try out your change: https://github.com/timblaktu/terraform-aws-eks/tree/568-redux In addition to your changes I had to replace this line in locals {
cluster_sg_name = coalesce(var.cluster_security_group_name, "${var.cluster_name}-cluster")
create_cluster_sg = local.create && var.create_cluster_security_group
#cluster_security_group_id = local.create_cluster_sg ? aws_security_group.cluster[0].id : var.cluster_security_group_id
cluster_security_group_id = local.create_cluster_sg ? coalescelist(aws_security_group.cluster[*].id, [""])[0] : var.cluster_security_group_idBut know the ❯ terraform -chdir=tf-aws-eks-blueprint destroy -var-file=../cluster-k8s-sbx/variables.tfvars -target="module.eks_blueprints" -auto-approve
data.aws_subnets.private: Reading...
module.eks_blueprints.module.aws_eks.module.kms.data.aws_partition.current: Reading...
module.eks_blueprints.data.aws_region.current: Reading...
module.eks_blueprints.module.aws_eks.data.aws_partition.current: Reading...
data.aws_vpc.selected: Reading...
module.eks_blueprints.data.aws_partition.current: Reading...
module.eks_blueprints.module.aws_eks.module.kms.data.aws_caller_identity.current: Reading...
module.eks_blueprints.module.aws_eks.data.aws_caller_identity.current: Reading...
module.eks_blueprints.module.aws_eks.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints.data.aws_region.current: Read complete after 0s [id=eu-central-1]
module.eks_blueprints.module.aws_eks.module.kms.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints.data.aws_partition.current: Read complete after 0s [id=aws]
module.eks_blueprints.data.aws_caller_identity.current: Reading...
module.eks_blueprints.module.aws_eks.data.aws_iam_policy_document.assume_role_policy[0]: Reading...
module.eks_blueprints.module.aws_eks.data.aws_iam_policy_document.assume_role_policy[0]: Read complete after 0s [id=276XXXXXXXX]
data.aws_subnets.private: Read complete after 0s [id=eu-central-1]
data.aws_vpc.selected: Read complete after 0s [id=vpc-0adXXXXXXXXXXXXXX]
module.eks_blueprints.module.aws_eks.data.aws_caller_identity.current: Read complete after 0s [id=4842XXXXXXX]
module.eks_blueprints.data.aws_caller_identity.current: Read complete after 0s [id=4842XXXXXXXX]
module.eks_blueprints.module.aws_eks.module.kms.data.aws_caller_identity.current: Read complete after 0s [id=4842XXXXXXXX]
module.eks_blueprints.data.aws_iam_session_context.current: Reading...
module.eks_blueprints.data.aws_iam_session_context.current: Read complete after 1s [id=arn:aws:sts::48XXXXXXXXXX:assumed-role/XXXXXXXXXXXXXXXXXXXXXXXXXX]
module.eks_blueprints.data.aws_iam_policy_document.eks_key: Reading...
module.eks_blueprints.data.aws_iam_policy_document.eks_key: Read complete after 0s [id=1297XXXXXXXXXX]
module.eks_blueprints.module.kms[0].aws_kms_key.this: Refreshing state... [id=f4XXXXXXXXXXXXXXX]
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an error message.
╵
╷
│ Error: "name" length must be between 1-100 characters: ""
│
│ with data.aws_eks_cluster.cluster,
│ on data.tf line 8, in data "aws_eks_cluster" "cluster":
│ 8: name = module.eks_blueprints.eks_cluster_id
│
╵
╷
│ Error: "name" doesn't comply with restrictions ("^[0-9A-Za-z][A-Za-z0-9\\-_]+$"): ""
│
│ with data.aws_eks_cluster.cluster,
│ on data.tf line 8, in data "aws_eks_cluster" "cluster":
│ 8: name = module.eks_blueprints.eks_cluster_id
│
╵
╷
│ Error: name must not be empty, got
│
│ with data.aws_eks_cluster_auth.cluster,
│ on data.tf line 12, in data "aws_eks_cluster_auth" "cluster":
│ 12: name = module.eks_blueprints.eks_cluster_id
│
╵
╷
│ Error: Invalid URL
│
│ with module.eks_blueprints.module.aws_eks.data.tls_certificate.this[0],
│ on .terraform/modules/eks_blueprints.aws_eks/main.tf line 206, in data "tls_certificate" "this":
│ 206: url = coalescelist(aws_eks_cluster.this[*].identity[0].oidc[0].issuer, [""])[0]
│
│ URL "" contains no host
╵
╷
│ Error: "name" length must be between 1-100 characters: ""
│
│ with module.eks_blueprints.data.aws_eks_cluster.cluster[0],
│ on .terraform/modules/eks_blueprints/data.tf line 7, in data "aws_eks_cluster" "cluster":
│ 7: name = module.aws_eks.cluster_id
│
╵
╷
│ Error: "name" doesn't comply with restrictions ("^[0-9A-Za-z][A-Za-z0-9\\-_]+$"): ""
│
│ with module.eks_blueprints.data.aws_eks_cluster.cluster[0],
│ on .terraform/modules/eks_blueprints/data.tf line 7, in data "aws_eks_cluster" "cluster":
│ 7: name = module.aws_eks.cluster_id
│
╵I am not sure if this is the right direction. I will leave the TF in this broken state, so that I can test additional things. Edit: I just rolled back all changes and deployed an older state to a new cluster, which works flawlessly - including destroy. Here is the full list of versions:
I am now introducing the changes step by step to identify the source. This takes quite some time. |
|
@ayeks thanks for testing and reporting! I can confirm I'm getting same behavior as you. Can you confirm whether you are using terraform 1.3.4? After playing whack-a-mole on this problem for a couple days and still finding it popping up even in modules I haven't changed, I have come to realize that much of this is a known issue in terraform 1.3.4. See my comment for details. |
|
@timblaktu Oh my god. I just switched to terraform 1.3.3 from 1.3.4 and ran the destroy command locally without any issues. If you ever come to Nuremberg, I will definitely buy you a beer for that hint. Thank you so much! I will run our end to end testing pipeline with the pinned version, just to make sure that I have not missed anything. What a crazy bug. Dependency hell at its finest. Edit: end2end test worked fine. I think this issue can be closed. Thanks again! |
|
it looks like fixes have been made to Terraform core now: |
|
Awesome. Let me test again using Terraform 1.35 (released with these 2 fixes) and terraform-aws-eks-blueprints v4.17.0 |
|
Now I am getting different errors ( Error: Unsupported attribute) on final terraform destroy terraform-aws-eks-blueprints v4.17.0 |
|
@armujahid I'm using eks_blueprints as well and saw those same errors on my first destroy sequence following the upgrade to 1.3.5. FWIW, those went away after I (in an unknown order):
Since, I have run my end-to-end lifecycle pipeline (apply sequence, some testing, destroy sequence) a few times with no errors. I should note here that before @jbardin merged that PR that closed the 1.3.4 destroy refresh issue I linked earlier, he had recommended to add p.s. @ayeks thanks for the offer - may take you up on that sometime - my wife and son just got their German citizenship and we're considering moving to Bavaria in a couple years. :-) |
|
Tested again by creating and destroying fresh cluster using terraform v.1.3.6 terraform-aws-eks-blueprints v4.17.0 Should I create a new issue? Since invalid index issue is no longer occurring since terraform v.1.3.5 and this seems like a different issue |
|
@armujahid I agree that that seems like a separate issue from the original indexing issue. |
|
Created a separate issue for "Error: Unsupported attribute". Closing since original issue is no longer reproducible. Thanks everyone :) |
Description
Deploy karpenter example with released version using v4.15.0 (git checkout tags/v4.15.0)
Destroy cluster in multiple steps.
This last step (terraform destroy -auto-approve) throws this error
Versions
Module version [Required]: v4.15.0
Terraform version:
Reproduction Code [Required]
Steps to reproduce the behavior:
At this point cluster will be available
Expected behaviour
Cluster should destroy without any error and
terraform.tfstateshouldn't have any residual resourceActual behaviour
error mentioned above is thrown and
terraform.tfstatestill has some residual resources although the cluster has been removed along with the VPCTerminal Output Screenshot(s)
Additional context
Note that I am migrating from v4.5.0 to 4.15.0 and I am getting similar Invalid index issues during cluster tear down. I have other helm releases in my project but this minimal, reproducible example is also exhibiting similar behavior. Examples of some errors that I am getting in my project are
I am not sure how to recover from this situation.
The text was updated successfully, but these errors were encountered: