Umbrella Issue: Kubernetes Infrastructure Tech Debt

[upodroid]

This is a bunch of technical debt in our Infra that needs to be resolved in the future.

IaC(Infrastructure as Code)

Beta Give feedback

1. Manage DNS via Terraform instead of Octodns
2. Rewrite the bash scripts at https://github.com/kubernetes/k8s.io/tree/main/infra/gcp/bash as terraform
3. Enable Terraform self-service for changes with Atlantis

Kubernetes

Beta Give feedback

1. Recreate the GKE clusters on custom networks
2. Create a doc with IP allocations for our infra across multiple cloud platforms to avoid IP overlap issues. We need to connect our EKS build cluster to a VMware Private Cloud network so our IP space needs to be properly carved out and documented.
3. eks-prow-build-cluster: Replace FluxCD with ArgoCD #6442
   0 of 5
   area/infra area/infra/aws priority/important-soon sig/k8s-infra +4
   
4. Consolidate all the Kubernetes manifests that will be used by ArgoCD in one location
5. Harden EKS/GKE Clusters #5110
   area/infra area/infra/aws area/infra/gcp priority/important-soon sig/k8s-infra +5

Please edit this issue with other technical debts that we have.

[BenTheElder]

Manage DNS via Terraform instead of Octodns

So taking this as an example, I don't think we should just declare this tech-debt without more details outlining why this is bad and why the alternative is better.

AFAICT our DNS utils are working great, they're relatively small and yaml is familiar to our community at large more than terraform.

This tool doesn't have any external state and does a reliable, kubernetes style "reconcile desired state versus the world", and we have some other neat tricks built in e.g. to staging deploy and test that first.

Rewrite the bash scripts at https://github.com/kubernetes/k8s.io/tree/main/infra/gcp/bash as terraform

Similarly, some of the bash scripts may be tech debt, but terraform isn't free to maintain either and we haven't had problems with most of these.

In general this checklist seems pretty opinionated without justifications.