Going forward, how should we deploy non-default EKS Add-Ons?

[ntwkninja]

The current EKS Blueprints that we use for Add-Ons is making changes in v5. Originally, the plan was to migrate to terraform-aws-eks-addons at the terraform-aws-modules GH org (complimentary module to terraform-aws-eks). Recently, EKS blueprints has decided to move to terraform-aws-eks-blueprints-addon in the aws-ia GH org. Since we're going to need to refactor the add-ons, I think this is a good opportunity to set up EKS Add-Ons how we want them to be going forward.

Default EKS Add-Ons (not part of this poll) are: 1.) kubeproxy, 2.) coredns & 3.) vpc-cni

Edit: Blueprints was planning to deprecate the current repo and migrate their add-ons to a new repo / org called terraform-aws-eks-addons but pivoted to standing up a new repo in the same aws-ia GH org.

[RothAndrew]

I think it is appropriate to do these types of addons as part of the terraform deployment, so my vote is for either the new EKS blueprints direction, or the original plan to use terraform-aws-eks-addons, with some kind of bakeoff/AoA to choose between the two. Even if we only spend 5 minutes on an AoA, I think as long as we document why we picked one over the other that works for me.

[ntwkninja]

I think it is appropriate to do these types of addons as part of the terraform deployment, so my vote is for either the new EKS blueprints direction, or the original plan to use terraform-aws-eks-addons, with some kind of bakeoff/AoA to choose between the two. Even if we only spend 5 minutes on an AoA, I think as long as we document why we picked one over the other that works for me.

we'll need to ensure kyverno registry exceptions are in place for things like calico (or try switching to the tigera-operator iron bank container via the terraform helm config.

also, need to consider how default storage classes will be handled (fits into this topic)

[ntwkninja]

my vote is for either the new EKS blueprints direction, or the original plan to use terraform-aws-eks-addons, with some kind of bakeoff/AoA to choose between the two.

Added an edit to the original comment to clarify terraform-aws-eks-addons.

[brianrexrode]

I would prefer to keep the add-ons as part of the terraform IaC deployment. What I've been seeing in Govcloud deployments is that the kubernetes add-on for calico fails pretty much every time on the initial deployment due to a lag in the amount of time it takes the kube-api to become available once the eks cluster is Active. This requires an additional terraform apply of the eks module. So as a user of the delivery-aws-iac, I hope whichever solution is decided upon will address this issue while not requiring too many additional steps.