Terraform.io to EKS "Error: Kubernetes cluster unreachable" #400

eeeschwartz · 2020-02-07T15:33:37Z

Terraform Version

0.12.19

Affected Resource(s)

helm_release

Terraform Configuration Files

locals {
 kubeconfig = <<KUBECONFIG
apiVersion: v1
clusters:
- cluster:
    server: ${aws_eks_cluster.my_cluster.endpoint}
    certificate-authority-data: ${aws_eks_cluster.my_cluster.certificate_authority.0.data}
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "${aws_eks_cluster.my_cluster.name}"
KUBECONFIG
}

resource "local_file" "kubeconfig" {
  content  = local.kubeconfig
  filename = "/home/terraform/.kube/config"
}

resource "null_resource" "custom" {
  depends_on    = [local_file.kubeconfig]

  # change trigger to run every time
  triggers = {
    build_number = "${timestamp()}"
  }

  # download kubectl
  provisioner "local-exec" {
    command = <<EOF
      set -e

      curl -o aws-iam-authenticator https://amazon-eks.s3-us-west-2.amazonaws.com/1.14.6/2019-08-22/bin/linux/amd64/aws-iam-authenticator
      chmod +x aws-iam-authenticator
      mkdir -p $HOME/bin && cp ./aws-iam-authenticator $HOME/bin/aws-iam-authenticator && export PATH=$PATH:$HOME/bin

      echo $PATH

      aws-iam-authenticator

      curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
      chmod +x kubectl

      ./kubectl get po
    EOF
  }
}

resource "helm_release" "testchart" {
  depends_on    = [local_file.kubeconfig]
  name          = "testchart"
  chart         = "../../../resources/testchart"
  namespace     = "default"
}

Debug Output

Note that

kubectl get po reaches the cluster and reports "No resources found in default namespace."
while helm_release reports: "Error: Kubernetes cluster unreachable"
In earlier testing it errored with "Error: stat /home/terraform/.kube/config". Now that I write the local file to that location, it no longer errors. I assume that means it successfully reads the kube config.

https://gist.github.com/eeeschwartz/021c7b0ca66a1b102970f36c42b23a59

Expected Behavior

The testchart is applied

Actual Behavior

The helm provider is unable to reach the EKS cluster.

Steps to Reproduce

On terraform.io:

terraform apply

Important Factoids

Note that kubectl is able to communicate with the cluster. But something about the terraform.io environment, the .helm/config, or the helm provider itself renders the cluster unreachable.

Note of Gratitude

Thanks for all the work getting helm 3 support out the door. Holler if I'm missing anything obvious or can help diagnose further.

The text was updated successfully, but these errors were encountered:

eeeschwartz · 2020-02-07T19:14:27Z

The token auth configuration below ultimately worked for me. Perhaps this should be the canonical approach for Terraform Cloud -> EKS, rather than using ~/.kube/config

provider "aws" {
  region = "us-east-1"
}

data "aws_eks_cluster_auth" "cluster-auth" {
  depends_on = [aws_eks_cluster.my_cluster]
  name       = aws_eks_cluster.my_cluster.name
}

provider "helm" {
  alias = "my_cluster"
  kubernetes {
    host                   = aws_eks_cluster.my_cluster.endpoint
    cluster_ca_certificate = base64decode(aws_eks_cluster.my_cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster-auth.token
    load_config_file       = false
  }
}

resource "helm_release" "testchart" {
  provider  = helm.my_cluster
  name       = "testchart"
  chart      = "../../../resources/testchart"
  namespace  = "default"
}

kinihun · 2020-02-29T22:23:02Z

I don't see how ths could possibly work, with Helm 3, it seems to be completely broken.
Below is my configuration and I can't connect to the cluster.
My kubernetes provider works but not the kubernetes block within Helm which has the same settings.

data "aws_eks_cluster" "cluster" {
  name = "foobar"
}

data "aws_eks_cluster_auth" "cluster" {
  name = "foobar"
}

provider "kubernetes" {
  version                = "1.10.0"
  host                   = data.aws_eks_cluster.cluster.endpoint
  cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
  token                  = data.aws_eks_cluster_auth.cluster.token
  load_config_file       = false
}

provider "helm" {
  version                = "1.0.0"

  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.cluster.token
    load_config_file       = false
  }
}```

venky999 · 2020-03-03T17:52:26Z

seeing same issue

kharandziuk · 2020-03-07T07:42:58Z

@eeeschwartz can confirm it's working with newly created cluster. Here a sample configuration to prove it.

From other perspective with already created cluster I see the same issue and debug=true doesn't help at all

vfiset · 2020-03-20T17:49:26Z

anyone found a workaround yet ?

kharandziuk · 2020-03-20T18:58:40Z

@vfiset I believe there is no workaround. It's just an issue with policies(my guess).

The actual provider doesn't give you enough debug information. So, you will probably need to run helm install manually to find an issue.

rgardam · 2020-03-30T02:53:43Z

My guess is that the aws-auth config map is blocking access. In the example that @kharandziuk has show here there's no aws-auth configmap defined. Also it's worth noting that the usage of helm here is in the same terraform run as the eks run which means that the default credentials for eks are the ones being used to deploy helm.

I have a fairly complicated setup where i'm assuming roles between the different stages of the EKS cluster deployment.

https://aws.amazon.com/premiumsupport/knowledge-center/amazon-eks-cluster-access/
Have a look at how you have to configure aws-auth to allow access to your cluster.

leoddias · 2020-06-09T13:56:52Z

Seeing same issue using Helm3. My tf looks like as @kinihun ...
Its happens in the first run of "terraform apply", when I try to exec again everything goes well.

netflash · 2020-06-30T13:01:54Z

I have created deployed a helm chart via helm provider ages ago. It works fine, I can change things here and there, etc.
Today I wanted to "migrate" a standalone-deployed helm chart to be managed under terraform. So when I tried to run terraform import helm_release.chart namespace/chart, I've got this error.

privomark · 2020-07-15T22:40:02Z

Seeing same issue using Helm3. My tf looks like as @kinihun ...
Its happens in the first run of "terraform apply", when I try to exec again everything goes well.

Same behavior as @leoddias . I've added the helm provider reference to k8s cluster and even did some local exec to switch contexts to the correct one. Also getting dial tcp i/o timeout errors in addition that all magically work on the 2nd apply.

mbelang · 2020-07-22T14:07:24Z

Anyone found a workaround here? I still get this: Error: Kubernetes cluster unreachable: invalid configuration: client-key-data or client-key must be specified for to use the clientCert authentication method.

mbelang · 2020-07-22T16:22:24Z

Works after the first apply because of this but the next plan, even if nothing changes will still re-generate the token.

------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
 <= read (data resources)

Terraform will perform the following actions:

  # data.aws_eks_cluster_auth.auth will be read during apply
  # (config refers to values not yet known)
 <= data "aws_eks_cluster_auth" "auth"  {
      + id    = (known after apply)
      + name  = "my_cluster"
      + token = (sensitive value)
    }

Plan: 0 to add, 0 to change, 0 to destroy.

------------------------------------------------------------------------

BrianMusson · 2020-07-29T20:56:46Z

Same issue here with Helm3

provider "helm" {
  version = "~> 1.2.3"

  kubernetes {
    load_config_file       = false
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)
    token                  = data.aws_eks_cluster_auth.kubernetes_token.token
  }
}

ma-caylent · 2020-09-17T16:30:29Z

I have created deployed a helm chart via helm provider ages ago. It works fine, I can change things here and there, etc.
Today I wanted to "migrate" a standalone-deployed helm chart to be managed under terraform. So when I tried to run terraform import helm_release.chart namespace/chart, I've got this error.

same issue here

jvanwygerden · 2020-09-17T21:17:56Z

Is there a fix available for this issue, by chance (or a timeline) ? @ HashiCorp team

voron · 2020-09-18T15:00:27Z

My workaround is to refresh data.aws_eks_cluster_auth before apply

terraform refresh -target=data.aws_eks_cluster_auth.cluster
terraform apply -target=helm_release.helm-operator -refresh=false

alexsomesan · 2020-09-18T15:06:29Z

You can have the token automatically refreshed if you configure the provider with an exec block with the aws cli or or the aws-iam-Authenticator. Have a look at the Kubeconfig in EKS docs for syntax details.

On Fri, Sep 18, 2020 at 6:00 PM Alex Vorona ***@***.***> wrote: My workaround is to refresh data.aws_eks_cluster_auth before apply terraform refresh -target=data.aws_eks_cluster_auth.cluster terraform apply -target=helm_release.helm-operator -refresh=false — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#400 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIL5G3MIR3GLPJHLISOTZ3SGNYZ7ANCNFSM4KRQNFKA> . --

— Sent from my phone.

voron · 2020-09-20T08:54:22Z

You can have the token automatically refreshed if you configure the
provider with an exec block with the aws cli or or the
aws-iam-Authenticator.

It doesn't change anything in my tests with terraform apply -refresh=false. And it doesn't required when data.aws_eks_cluster_auth.cluster is refreshed by terraform configuration.

kwahsog · 2020-10-07T11:17:36Z

@voron thanks! The refresh fix was the only thing that worked for me.

netflash · 2020-10-16T22:02:39Z

Tried import with 1.3.2

Got this

terraform import helm_release.jenkins jenkins/jenkins
helm_release.jenkins: Importing from ID "jenkins/jenkins"...

Error: Kubernetes cluster unreachable: the server has asked for the client to provide credentials

but terraform plan works fine (my terraform config that has other helm resources)

derkoe · 2020-11-05T12:19:09Z

We had this problem with an RBAC enabled cluster. In this case the token from the cluster creation did not have enough permissions.

avodaqstephan · 2020-11-12T14:10:36Z

Seeing same issue using Helm3. My tf looks like as @kinihun ...
Its happens in the first run of "terraform apply", when I try to exec again everything goes well.

Same here.

biswa-r-singh · 2020-12-02T08:09:00Z

I followed the instruction #400 (comment) provided by @eeeschwartz in this thread. It would fail for the first apply and work second time. The only thing that i missed was was adding "depends_on = [aws_eks_cluster.my_cluster]" to the data resource as mentioned in the code snippet. Once i added it started working. I created and destroyed the deployment multiple times and it worked.

data "aws_eks_cluster_auth" "cluster-auth" {
// Add the depends_on
name = aws_eks_cluster.my_cluster.name
}

jroberts235 · 2021-02-25T22:06:03Z

Switching from 2.0.2 to version 1.3.2 of the Helm provider fixed our config issues.

ghassencherni · 2021-03-03T11:23:36Z

After unsetting ENV vars for kubectl , that were pointing to the old cluster everything worked:

unset KUBECONFIG
unset KUBE_CONFIG_PATH

Not sure why helm provider reads those vars if following setup was used:

provider "helm" {  
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority[0].data)
    token                  = data.aws_eks_cluster_auth.cluster.token  
  }
}

Using 2.0.2 provider versions that don't have "load_config_file" argument available anymore.

dak1n1 · 2021-04-03T15:24:41Z

I'm going to close this issue since the OP has a solution, and since we have several similar issues open already between the Kubernetes and Helm providers. We are continuing to work on the authentication workflow to make configuration easier. (These are the next steps toward fixing it, if anyone is curious: hashicorp/terraform-provider-kubernetes#1141 and hashicorp/terraform-plugin-sdk#727).

formatlos · 2021-04-21T15:36:10Z

I do have the same problem, I've tried all of the mentioned solutions but it doesn't seem to pick up the token properly.

Terraform v0.14.4
hashicorp/helm v2.0.3

this is my config:

data "aws_eks_cluster" "cluster" {
  name = module.eks.cluster_id
}

data "aws_eks_cluster_auth" "cluster" {
  name = module.eks.cluster_id
}

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.cluster.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.cluster.certificate_authority.0.data)

    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster_auth.cluster.id]
      command     = "aws"
    }
  }
}

any thoughts?

goetzc · 2021-05-03T16:03:38Z

@formatlos Running on Terraform Cloud, using Terraform 0.15.1 and Helm provider 2.1.2, your solution with exec authentication works for me. Just changed on the args to get the cluster name instead of the id.

provider "helm" {
  kubernetes {
    host                   = data.aws_eks_cluster.eks.endpoint
    cluster_ca_certificate = base64decode(data.aws_eks_cluster.eks.certificate_authority.0.data)

    exec {
      api_version = "client.authentication.k8s.io/v1alpha1"
      command     = "aws"
      args        = ["eks", "get-token", "--cluster-name", data.aws_eks_cluster.eks.name]
    }
  }
}

ghost · 2021-05-04T05:10:00Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 hashibot-feedback@hashicorp.com. Thanks!

eeeschwartz mentioned this issue Feb 10, 2020

home/.kube/config: no such file or directory #402

Closed

aktech mentioned this issue May 13, 2020

Testing AWS QHub support Quansight/qhub-ops#16

Closed

aareet added theme/errors bug needs-investigation labels Jul 2, 2020

dak1n1 added the size/M label Jul 15, 2020

aLekSer mentioned this issue Sep 4, 2020

Docs and TF: Update EKS Kubernetes version to use 1.16 googleforgames/agones#1790

Merged

lpossamai mentioned this issue Feb 19, 2021

Error: Kubernetes cluster unreachable with helm 3.0 k3s-io/k3s#1126

Closed

dak1n1 added the theme/auth label Mar 22, 2021

dak1n1 closed this as completed Apr 3, 2021

ashtonian mentioned this issue Apr 22, 2021

Kubernetes cluster unreachable: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable terraform-aws-modules/terraform-aws-eks#1234

Closed

ghost locked as resolved and limited conversation to collaborators May 4, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Terraform.io to EKS "Error: Kubernetes cluster unreachable" #400

Terraform.io to EKS "Error: Kubernetes cluster unreachable" #400

eeeschwartz commented Feb 7, 2020 •

edited

eeeschwartz commented Feb 7, 2020 •

edited

kinihun commented Feb 29, 2020 •

edited

venky999 commented Mar 3, 2020

kharandziuk commented Mar 7, 2020

vfiset commented Mar 20, 2020

kharandziuk commented Mar 20, 2020

rgardam commented Mar 30, 2020

leoddias commented Jun 9, 2020

netflash commented Jun 30, 2020

privomark commented Jul 15, 2020

mbelang commented Jul 22, 2020

mbelang commented Jul 22, 2020

BrianMusson commented Jul 29, 2020 •

edited

ma-caylent commented Sep 17, 2020

jvanwygerden commented Sep 17, 2020 •

edited

voron commented Sep 18, 2020

alexsomesan commented Sep 18, 2020 via email

voron commented Sep 20, 2020

kwahsog commented Oct 7, 2020

netflash commented Oct 16, 2020

derkoe commented Nov 5, 2020

avodaqstephan commented Nov 12, 2020

biswa-r-singh commented Dec 2, 2020 •

edited

jroberts235 commented Feb 25, 2021

ghassencherni commented Mar 3, 2021 •

edited

dak1n1 commented Apr 3, 2021

formatlos commented Apr 21, 2021

goetzc commented May 3, 2021

ghost commented May 4, 2021

Terraform.io to EKS "Error: Kubernetes cluster unreachable" #400

Terraform.io to EKS "Error: Kubernetes cluster unreachable" #400

Comments

eeeschwartz commented Feb 7, 2020 • edited

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Debug Output

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

Note of Gratitude

eeeschwartz commented Feb 7, 2020 • edited

kinihun commented Feb 29, 2020 • edited

venky999 commented Mar 3, 2020

kharandziuk commented Mar 7, 2020

vfiset commented Mar 20, 2020

kharandziuk commented Mar 20, 2020

rgardam commented Mar 30, 2020

leoddias commented Jun 9, 2020

netflash commented Jun 30, 2020

privomark commented Jul 15, 2020

mbelang commented Jul 22, 2020

mbelang commented Jul 22, 2020

BrianMusson commented Jul 29, 2020 • edited

ma-caylent commented Sep 17, 2020

jvanwygerden commented Sep 17, 2020 • edited

voron commented Sep 18, 2020

alexsomesan commented Sep 18, 2020 via email

voron commented Sep 20, 2020

kwahsog commented Oct 7, 2020

netflash commented Oct 16, 2020

derkoe commented Nov 5, 2020

avodaqstephan commented Nov 12, 2020

biswa-r-singh commented Dec 2, 2020 • edited

jroberts235 commented Feb 25, 2021

ghassencherni commented Mar 3, 2021 • edited

dak1n1 commented Apr 3, 2021

formatlos commented Apr 21, 2021

goetzc commented May 3, 2021

ghost commented May 4, 2021

eeeschwartz commented Feb 7, 2020 •

edited

eeeschwartz commented Feb 7, 2020 •

edited

kinihun commented Feb 29, 2020 •

edited

BrianMusson commented Jul 29, 2020 •

edited

jvanwygerden commented Sep 17, 2020 •

edited

biswa-r-singh commented Dec 2, 2020 •

edited

ghassencherni commented Mar 3, 2021 •

edited