[E2E] EKS jobs are failing

[barkbay]

EKS jobs are failing with the following error:

03:03:58  panic: can't determine Kubernetes version, err invalid character 'K' looking for beginning of value
03:03:58  
03:03:58  goroutine 1 [running]:
03:03:58  github.com/elastic/cloud-on-k8s/test/e2e/cmd/run.getKubernetesVersion(0x1aedc14?)
03:03:58  	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:226 +0x2f4
03:03:58  github.com/elastic/cloud-on-k8s/test/e2e/cmd/run.(*helper).initTestContext(0xc00084c700)
03:03:58  	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:185 +0x265
03:03:58  github.com/elastic/cloud-on-k8s/test/e2e/cmd/run.doRun({{0xc00070df60, 0x2, 0x2}, {0x7ffe2128aa2d, 0x37}, {0x7ffe2128aaf7, 0x5}, {0x7ffe2128ab14, 0x17}, {0x0, ...}, ...})
03:03:58  	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/run.go:90 +0x663
03:03:58  github.com/elastic/cloud-on-k8s/test/e2e/cmd/run.Command.func2(0xc000838780?, {0x1adffdf?, 0x12?, 0x12?})
03:03:58  	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/run/command.go:74 +0x85
03:03:58  github.com/spf13/cobra.(*Command).execute(0xc000838780, {0xc00056e5a0, 0x12, 0x12})
03:03:58  	/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x67c
03:03:58  github.com/spf13/cobra.(*Command).ExecuteC(0xc000838500)
03:03:58  	/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974 +0x3b4
03:03:58  github.com/spf13/cobra.(*Command).Execute(...)
03:03:58  	/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902
03:03:58  main.main()
03:03:58  	/go/src/github.com/elastic/cloud-on-k8s/test/e2e/cmd/main.go:32 +0xfb

I think the root cause is the following line inserted at the beginning of the output of the kubectl command:

Kubeconfig user entry is using deprecated API version client.authentication.k8s.io/v1alpha1. Run 'aws eks update-kubeconfig' to update.

Might be related to https://github.com/elastic/k8s-dev/issues/362

[barkbay]

Closing as #5669 seems to workaround the problem for now:

[barkbay]

Looks like it is no more deprecated:

03:09:09  Error: failed to create kubernetes client: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

[thbkrkr]

We are using kubectl v1.19.16 in the CI image and the bug happens with kubectl v1.24.0, so what's going on? #5662 upgraded the k8s.io/client-go dependency to v0.24.0.

Despite we get a recent aws-cli version (2.7.0), we are using eksctl (0.74.0) to write the .kubeconfigfile that uses the now invalid apiVersion client.authentication.k8s.io/v1alpha1. I tested with the latest eksctl version (0.97.0) and it is not yet updated.

One way to fix this issue might be to use aws eks update-kubeconfig --name {{.ClusterName}} --region {{.Region}} instead of eksctl utils write-kubeconfig --cluster {{.ClusterName}} --region {{.Region}} to get a valid kubeconfig that works with k8s.io/client-go v0.24.0.