EKSA vSphere package controller pod x509 certificate signed by unknown authority

[saiteja313]

What happened:

• Package controller pod on EKSA vSphere cluster failling with X509 certificate signed by unknown authority error.
• Tried re-installing the package controller [1] using below commands and no luck. we still notice same error from pod logs.

helm uninstall -n eksa-packages eks-anywhere-packages
eksctl anywhere install packagecontroller -f <CLUSTER_CONFIG>.yaml

package controller pod logs

pulling package bundle: fetch manifest: Get "https://public.ecr.aws/v2/eks-anywhere/eks-anywhere-packages-bundles/manifests/v1-29-latest": x509: certificate signed by unknown authority

Helm list command output

 helm list --all-namespaces
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                                   APP VERSION               
eks-anywhere-packages   eksa-packages   1               2024-03-26 18:25:30.769847675 +0000 UTC deployed        eks-anywhere-packages-0.3.13-eks-a-60   v0.3.13-86cb2ba2e629eae21c79bca6bf78149e81f2527f       

Checked cert-manager and validate no errors.

> k logs cert-manager-848f9994fc-txvt9 -n cert-manager
I0320 20:27:42.651183       1 controller.go:251] "cert-manager/controller/build-context: configured acme dns01 nameservers" nameservers=["10.96.0.10:53"]
W0320 20:27:42.651245       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0320 20:27:42.653564       1 controller.go:72] "cert-manager/controller: enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]"
I0320 20:27:42.654030       1 controller.go:145] "cert-manager/controller: starting leader election"
I0320 20:27:42.654844       1 leaderelection.go:250] attempting to acquire leader lease kube-system/cert-manager-controller...
I0320 20:27:42.655194       1 controller.go:93] "cert-manager/controller: starting metrics server" address="[::]:9402"
I0320 20:27:42.655258       1 controller.go:138] "cert-manager/controller: starting healthz server" address="[::]:9403"
I0320 20:27:42.667318       1 leaderelection.go:260] successfully acquired lease kube-system/cert-manager-controller
I0320 20:27:42.668453       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-ca"
I0320 20:27:42.668906       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-acme"
I0320 20:27:42.669119       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="gateway-shim"
I0320 20:27:42.670341       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-selfsigned"
I0320 20:27:42.670348       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-venafi"
I0320 20:27:42.671524       1 controller.go:192] "cert-manager/controller: not starting controller as it's disabled" controller="certificatesigningrequests-issuer-vault"
I0320 20:27:42.671756       1 controller.go:215] "cert-manager/controller: starting controller" controller="ingress-shim"
I0320 20:27:42.671774       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-vault"
I0320 20:27:42.671787       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-key-manager"
I0320 20:27:42.671809       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-issuing"
I0320 20:27:42.671820       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-request-manager"
I0320 20:27:42.671831       1 controller.go:215] "cert-manager/controller: starting controller" controller="orders"
I0320 20:27:42.671842       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-acme"
I0320 20:27:42.671868       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-approver"
I0320 20:27:42.679670       1 controller.go:215] "cert-manager/controller: starting controller" controller="clusterissuers"
I0320 20:27:42.679696       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-venafi"
I0320 20:27:42.679724       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-ca"
I0320 20:27:42.679754       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificaterequests-issuer-selfsigned"
I0320 20:27:42.679772       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-metrics"
I0320 20:27:42.679796       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-readiness"
I0320 20:27:42.679816       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-trigger"
I0320 20:27:42.679844       1 controller.go:215] "cert-manager/controller: starting controller" controller="issuers"
I0320 20:27:42.679860       1 controller.go:215] "cert-manager/controller: starting controller" controller="challenges"
I0320 20:27:42.679877       1 controller.go:215] "cert-manager/controller: starting controller" controller="certificates-revision-manager"
E0320 21:15:15.230396       1 controller.go:134] "cert-manager/issuers: issuer in work queue no longer exists" err="[issuer.cert-manager.io](http://issuer.cert-manager.io/) \"eks-anywhere-packages-selfsigned-issuer\" not found"
I0320 21:15:51.505607       1 conditions.go:203] Setting lastTransitionTime for Certificate "eks-anywhere-packages-serving-cert" condition "Ready" to 2024-03-20 21:15:51.505579818 +0000 UTC m=+2888.887189117
I0320 21:15:51.515170       1 conditions.go:96] Setting lastTransitionTime for Issuer "eks-anywhere-packages-selfsigned-issuer" condition "Ready" to 2024-03-20 21:15:51.515164279 +0000 UTC m=+2888.896773587
E0326 18:25:12.590651       1 controller.go:134] "cert-manager/issuers: issuer in work queue no longer exists" err="[issuer.cert-manager.io](http://issuer.cert-manager.io/) \"eks-anywhere-packages-selfsigned-issuer\" not found"
I0326 18:25:31.462353       1 conditions.go:203] Setting lastTransitionTime for Certificate "eks-anywhere-packages-serving-cert" condition "Ready" to 2024-03-26 18:25:31.462345478 +0000 UTC m=+511068.843954778
I0326 18:25:31.482040       1 conditions.go:96] Setting lastTransitionTime for Issuer "eks-anywhere-packages-selfsigned-issuer" condition "Ready" to 2024-03-26 18:25:31.482033776 +0000 UTC m=+511068.863643081
E0326 18:33:23.986091       1 controller.go:134] "cert-manager/issuers: issuer in work queue no longer exists" err="[issuer.cert-manager.io](http://issuer.cert-manager.io/) \"eks-anywhere-packages-selfsigned-issuer\" not found"
I0326 18:33:41.750271       1 conditions.go:203] Setting lastTransitionTime for Certificate "eks-anywhere-packages-serving-cert" condition "Ready" to 2024-03-26 18:33:41.750264124 +0000 UTC m=+511559.131873435
I0326 18:33:41.767384       1 conditions.go:96] Setting lastTransitionTime for Issuer "eks-anywhere-packages-selfsigned-issuer" condition "Ready" to 2024-03-26 18:33:41.767376284 +0000 UTC m=+511559.148985588

There are four secrets as Volume mount to Package controller pod

kubectl get secret -n eksa-packages webhook-server-cert -o yaml
kubectl get secret -n eksa-packages registry-mirror-cred -o yaml
ekubectl get secret -n eksa-packages ecr-token -o yaml
kubectl get secret -n eksa-packages aws-secret -o yaml

• webhook-server-cert secret got ca.crt, tls.crt and tls.key.crt files data in it. ca.crt shows 3 months validity for the certificate. tls.key.crt fails with below error when tried to read it.
• ecr-token got decoded using base64 and able to see json data needed to authenticate with ECR.

openssl x509 -in webhook-server-cert.tls.key.crt --noout --text

Could not read certificate from webhook-server-cert.tls.key.crt
800B5C3D057F0000:error:1608010C:STORE routines:ossl_store_handle_load_result:unsupported:crypto/store/store_result.c:151:

---

Docker pull works fine from Admin machine. below is output for manual docker pull.

aws sts get-caller-identity
{
    "UserId": "REDACTED",
    "Account": "REDACTED",
    "Arn": "arn:aws:iam::REDACTED:user/service/eksa-curated-package-user"
}
aws ecr get-login-password | docker login --username AWS --password-stdin [REDACTED.dkr.ecr.us-west-2.amazonaws.com](http://REDACTED.dkr.ecr.us-west-2.amazonaws.com/)

Error response from daemon: login attempt to https://REDACTED.dkr.ecr.us-west-2.amazonaws.com/v2/ failed with status: 400 Bad Request
> aws ecr get-login-password | docker login --username AWS --password-stdin [REDACTED.dkr.ecr.us-east-1.amazonaws.com](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/)

WARNING! Your password will be stored unencrypted in /home/REDACTED/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

> docker pull [REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074)

v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074: Pulling from emissary-ingress/emissary
Digest: sha256:0429a4b17ea8b2845ec66de412640f599665aad52093ea62d5d564e788c9b5cc
Status: Image is up to date for [REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074)
[REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074](http://REDACTED.dkr.ecr.us-east-1.amazonaws.com/emissary-ingress/emissary:v3.5.1-bf70150bcdfe3a5383ec8ad9cd7eea801a0cb074)

---

References:
[1] https://anywhere.eks.amazonaws.com/docs/packages/packagecontroller/

What you expected to happen:

• eksctl anywhere install command should completed package controller installation correctly.

How to reproduce it (as minimally and precisely as possible):

• Create a EKSA vSphere cluster. package controller should be installed by default. uninstall the package controller and re-install using below command,

helm uninstall -n eksa-packages eks-anywhere-packages
eksctl anywhere install packagecontroller -f <CLUSTER_CONFIG>.yaml

• Validate the installation timeouts

Anything else we need to know?:

Environment:

• EKS Anywhere Release: v0.19.0
• EKS Distro Release: Kubernetes version 1.29