fully private cluster with lambda or dedicated bastion host as a proxy

[tomiszili]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

I want to create and manage a fully private EKS clusters without any additional VPC, VPC peering, Cloud9 instance etc.... The idea is came from the aws-quickstart-eks cloudformation templates.
Currently if you want to provision/modify anything inside the eks cluster - without the Cloud9 instance - (e.g.: edit the aws-auth configmap, deploying addons with helm charts or any kubernetes manifest...) you have to enable the public endpoint, then take the desired actions and then you have to disable the public endpoint again. As far as i know Terraform could not handle changes delayed in time to the same cluster, and consider the latest sdk action as the desired state, so this public endpoint on-off switching is unbearable.

Describe the solution you would like

One possible solution could be if the aws-auth, helm, and kubernetes providers could communicate through a lambda proxy attached to the same subnets as the EKS cluster. I know about the 15 minutes limitation (maybe step functions orchestrated lambdas?). I think the 15 minutes upper limit couldn't be a problem because in most cases only management addons (logging, monitoring, security, etc...) deployed with terraform.

Other possibility is to create a dedicated bastion host (in private subnet) to every cluster (e.g.: t4g.micro ec2) with ssm-agent installed due to open a ssm session with port forwarding (e.g.: for SSH tunneling purposes) to the EKS cluster's https endpoint. This tunneling acts as a proxy for the terraform providers.

Describe alternatives you have considered

Additional context

I'm not a terraform guru so i don't know if my idea could be implemented at all.
Hope for the best.

[rodrigobersa]

Hi @tomiszili,

Thanks for this request.

The examples/fully-private-cluster, suggests a deployment through a Cloud9 or an EC2 on an existing default VPC as bastion host, and already creates the VPC Peering and Route Tables to make the connection between the Bastion and EKS work.

You can adapt this to your scenario, changing the Source VPC for the Peering and Route Tables to your own private subnet ID. Another option is to deploy your bastion host together with your EKS Clusters in the same VPC, although this can lead you to deploy as many bastion hosts as the number of clusters you have, not sure if it's what you're looking for.

Also, it is possible to customize your aws-auth using the manage_aws_auth_configmap and aws_auth_roles variables from the Terraform EKS Module Inputs to customize the cluster access, like this example.

Let us know if any of this options can help.