Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[kube-prometheus-stack] prometheus-stack-kube-prom-operator Error : msg="http: TLS handshake error..." #4402

Open
Leeeuijooo opened this issue Mar 28, 2024 · 0 comments
Open

[kube-prometheus-stack] prometheus-stack-kube-prom-operator Error : msg="http: TLS handshake error..." #4402

Leeeuijooo opened this issue Mar 28, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@Leeeuijooo
Copy link

Describe the bug a clear and concise description of what the bug is.

msg="http: TLS handshake error from 10.0.226.54:36916: remote error: tls: bad certificate"

What's your helm version?

3.14.2

What's your kubectl version?

1.29.2

Which chart?

kube-prometheus-stack

What's the chart version?

latest

What happened?

I'm Deploying kube-prometheus-stack with ArgoCD Application using Amazon EKS

However, after deploying kube-prometheus-stack, the following error occurred in the prometheus-stack-kube-prom-operator pod.

level=info ts=2024-03-27T05:55:07.975606693Z caller=main.go:181 msg="Starting Prometheus Operator" version="(version=0.71.2, branch=refs/tags/v0.71.2, revision=af2014407bdc25c4fc2b26cd99c9655235ebdf88)"
level=info ts=2024-03-27T05:55:07.975666128Z caller=main.go:182 build_context="(go=go1.21.6, platform=linux/amd64, user=Action-Run-ID-7656327832, date=20240125-14:57:39, tags=unknown)"
level=info ts=2024-03-27T05:55:07.97568378Z caller=main.go:193 msg="namespaces filtering configuration " config="{allow_list=\"\",deny_list=\"\",prometheus_allow_list=\"\",alertmanager_allow_list=\"\",alertmanagerconfig_allow_list=\"\",thanosruler_allow_list=\"\"}"
level=info ts=2024-03-27T05:55:07.987141023Z caller=main.go:222 msg="connection established" cluster-version=v1.28.7-eks-b9c9ed7
level=info ts=2024-03-27T05:55:08.04542884Z caller=operator.go:333 component=prometheus-controller msg="Kubernetes API capabilities" endpointslices=true
level=info ts=2024-03-27T05:55:08.070368535Z caller=operator.go:319 component=prometheusagent-controller msg="Kubernetes API capabilities" endpointslices=true
level=warn ts=2024-03-27T05:55:08.091889063Z caller=server.go:160 msg="server TLS client verification disabled" client_ca_file=/etc/tls/private/tls-ca.crt err="stat /etc/tls/private/tls-ca.crt: no such file or directory"
level=info ts=2024-03-27T05:55:08.100261947Z caller=server.go:300 msg="starting secure server" address=[::]:10250 http2=false
level=info ts=2024-03-27T05:55:08.202585326Z caller=operator.go:390 component=prometheus-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.20309889Z caller=operator.go:311 component=alertmanager-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.202599338Z caller=operator.go:428 component=prometheusagent-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.202772554Z caller=operator.go:280 component=thanos-controller msg="successfully synced all caches"
level=info ts=2024-03-27T05:55:08.215416391Z caller=operator.go:987 component=prometheus-controller key=monitoring/prometheus-stack-kube-prom-prometheus msg="sync prometheus"
level=warn ts=2024-03-27T05:55:08.63480492Z caller=klog.go:106 component=k8s_client_runtime func=Warning msg="spec.template.spec.containers[1].ports[0]: duplicate port definition with spec.template.spec.initContainers[0].ports[0]"
level=info ts=2024-03-27T05:55:08.646201453Z caller=operator.go:987 component=prometheus-controller key=monitoring/prometheus-stack-kube-prom-prometheus msg="sync prometheus"
level=info ts=2024-03-27T05:55:09.183415232Z caller=operator.go:987 component=prometheus-controller key=monitoring/prometheus-stack-kube-prom-prometheus msg="sync prometheus"
ts=2024-03-27T05:56:07.223479031Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.226.54:36902: remote error: tls: bad certificate"
ts=2024-03-27T05:56:07.23267305Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.226.54:36916: remote error: tls: bad certificate"
ts=2024-03-27T05:56:07.262879691Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from 10.0.226.54:36930: remote error: tls: bad certificate"
ts=2024-03-27T05:56:07.267461471Z caller=stdlib.go:105 caller=server.go:3212 msg="http: TLS handshake error from

What you expected to happen?

When I connect to the domain, I am connected to the prometheus server and connected via 302 redirect to the /graph path.
However, a 404 error occurs on the redirected page.

How to reproduce it?

The IP (10.0.226.54) is an IP that does not exist in the EKS resource. In my opinion, it looks like the IP came in as NAT from an external source (client).

Also, it says that the tls-ca.crt file does not exist. Is there a way to create it or solve this problem?

Currently, one ALB is set up as Ingress on the Grafana and Prometheus servers.

Below is part of the values.yaml file for the prometheus-stack chart.

Enter the changed values of values.yaml?

grafana
...
  ingress:
    enabled: true
    ingressClassName: alb
    annotations: {
      alb.ingress.kubernetes.io/scheme: internet-facing,
      alb.ingress.kubernetes.io/target-type: ip,
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]',
      alb.ingress.kubernetes.io/certificate-arn: <AWS_CA>,
      alb.ingress.kubernetes.io/success-codes: 200-399,
      alb.ingress.kubernetes.io/load-balancer-name: monitoring-alb,
      alb.ingress.kubernetes.io/ssl-redirect: '443',
      alb.ingress.kubernetes.io/group.name: "monitoring"
      }
    labels: {}
    hosts:
      - grafana.help.store

    ## Path for grafana ingress
    path: /

    ## TLS configuration for grafana Ingress
    ## Secret must be manually created in the namespace
    ##
    tls: []
    # - secretName: grafana-general-tls
    #   hosts:
    #   - grafana.example.com

  # # To make Grafana persistent (Using Statefulset)
  # #
  persistence:
    enabled: true
  #   type: sts
    storageClassName: "gp2"
    accessModes:
      - ReadWriteOnce
    size: 20Gi
------------------------------------------------------------------------------
prometheus
  ingress:
    enabled: true
    ingressClassName: alb
    annotations: {
      alb.ingress.kubernetes.io/scheme: internet-facing,
      alb.ingress.kubernetes.io/target-type: ip,
      alb.ingress.kubernetes.io/listen-ports: '[{"HTTPS":443}, {"HTTP":80}]',
      alb.ingress.kubernetes.io/certificate-arn: <AWS_CA>,
      alb.ingress.kubernetes.io/success-codes: 200-399,
      alb.ingress.kubernetes.io/load-balancer-name: monitoring-alb,
      alb.ingress.kubernetes.io/group.name: "monitoring",
      alb.ingress.kubernetes.io/ssl-redirect: '443'
    }

    labels: {}

    ## Redirect ingress to an additional defined port on the service
    # servicePort: 8081
    hosts:
      - prometheus.help.store

    ## Paths to use for ingress rules - one path should match the prometheusSpec.routePrefix
    ##
    paths:
      - /

    ## For Kubernetes >= 1.18 you should specify the pathType (determines how Ingress paths should be matched)
    ## See https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#better-path-matching-with-path-types
    # pathType: ImplementationSpecific

    ## TLS configuration for Prometheus Ingress
    ## Secret must be manually created in the namespace
    ##
    tls: []
      # - secretName: prometheus-general-tls
      #   hosts:
      #     - prometheus.example.com

Enter the command that you execute and failing/misfunctioning.

I'm Deploying kube-prometheus-stack with ArgoCD using Amazon EKS

Anything else we need to know?

There is my EKS Resources

$ kubectl get all -n monitoring
NAME                                                       READY   STATUS    RESTARTS   AGE
pod/domain-exporter-64bf7f9949-w9b62                       1/1     Running   0          2d19h
pod/prometheus-prometheus-stack-kube-prom-prometheus-0     2/2     Running   0          19h
pod/prometheus-stack-grafana-6c4bf874c5-f4vqp              3/3     Running   0          19h
pod/prometheus-stack-kube-prom-operator-776c4b9f76-9cfpp   1/1     Running   0          19h
pod/prometheus-stack-kube-state-metrics-6d555c6cb9-pd5rc   1/1     Running   0          19h
pod/prometheus-stack-prometheus-node-exporter-jps67        1/1     Running   0          19h
pod/prometheus-stack-prometheus-node-exporter-xjp7s        1/1     Running   0          19h

NAME                                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
service/domain-exporter                             ClusterIP   172.20.157.164   <none>        9203/TCP            2d19h
service/prometheus-operated                         ClusterIP   None             <none>        9090/TCP            19h
service/prometheus-stack-grafana                    ClusterIP   172.20.75.61     <none>        80/TCP              19h
service/prometheus-stack-kube-prom-operator         ClusterIP   172.20.1.2       <none>        443/TCP             19h
service/prometheus-stack-kube-prom-prometheus       ClusterIP   172.20.245.196   <none>        9090/TCP,8080/TCP   19h
service/prometheus-stack-kube-state-metrics         ClusterIP   172.20.171.111   <none>        8080/TCP            19h
service/prometheus-stack-prometheus-node-exporter   ClusterIP   172.20.217.150   <none>        9100/TCP            19h

NAME                                                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/prometheus-stack-prometheus-node-exporter   2         2         2       2            2           kubernetes.io/os=linux   19h

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/domain-exporter                       1/1     1            1           2d19h
deployment.apps/prometheus-stack-grafana              1/1     1            1           19h
deployment.apps/prometheus-stack-kube-prom-operator   1/1     1            1           19h
deployment.apps/prometheus-stack-kube-state-metrics   1/1     1            1           19h

NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/domain-exporter-64bf7f9949                       1         1         1       2d19h
replicaset.apps/prometheus-stack-grafana-6c4bf874c5              1         1         1       19h
replicaset.apps/prometheus-stack-kube-prom-operator-776c4b9f76   1         1         1       19h
replicaset.apps/prometheus-stack-kube-state-metrics-6d555c6cb9   1         1         1       19h

NAME                                                                READY   AGE
statefulset.apps/prometheus-prometheus-stack-kube-prom-prometheus   1/1     19h
---------------------------------------------------------------------------------
$ kubectl get all -n kube-system
NAME                                                READY   STATUS    RESTARTS   AGE
pod/aws-load-balancer-controller-79574fb866-45g8s   1/1     Running   0          2d21h
pod/aws-load-balancer-controller-79574fb866-f2ctk   1/1     Running   0          2d21h
pod/aws-node-dkg7t                                  2/2     Running   0          2d22h
pod/aws-node-vq7hx                                  2/2     Running   0          2d22h
pod/coredns-56dfff779f-dbvbd                        1/1     Running   0          2d22h
pod/coredns-56dfff779f-hw7tv                        1/1     Running   0          2d22h
pod/ebs-csi-controller-66ccf4755d-vzl5p             6/6     Running   0          2d21h
pod/ebs-csi-controller-66ccf4755d-znphb             6/6     Running   0          2d21h
pod/ebs-csi-node-khnzv                              3/3     Running   0          2d21h
pod/ebs-csi-node-pw5bf                              3/3     Running   0          2d21h
pod/external-dns-7cfc59cdbf-g5tgp                   1/1     Running   0          41h
pod/kube-ops-view-9cc4bf44c-wfpbp                   1/1     Running   0          2d21h
pod/kube-proxy-5rpv7                                1/1     Running   0          2d22h
pod/kube-proxy-78c7d                                1/1     Running   0          2d22h
pod/s3-csi-node-b99sn                               3/3     Running   0          2d21h
pod/s3-csi-node-l2wj7                               3/3     Running   0          2d21h

NAME                                                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                   PORT(S)                        AGE
service/aws-load-balancer-webhook-service                    ClusterIP      172.20.238.252   <none>                                                                        443/TCP                        2d21h
service/kube-dns                                             ClusterIP      172.20.0.10      <none>                                                                        53/UDP,53/TCP                  2d22h
service/kube-ops-view                                        LoadBalancer   172.20.93.114    a9de573cf32c240c189921c55595c07e-484039070.ap-northeast-2.elb.amazonaws.com   8080:31263/TCP                 2d21h
service/prometheus-stack-kube-prom-coredns                   ClusterIP      None             <none>                                                                        9153/TCP                       19h
service/prometheus-stack-kube-prom-kube-controller-manager   ClusterIP      None             <none>                                                                        10257/TCP                      19h
service/prometheus-stack-kube-prom-kube-etcd                 ClusterIP      None             <none>                                                                        2381/TCP                       19h
service/prometheus-stack-kube-prom-kube-proxy                ClusterIP      None             <none>                                                                        10249/TCP                      19h
service/prometheus-stack-kube-prom-kube-scheduler            ClusterIP      None             <none>                                                                        10259/TCP                      19h
service/prometheus-stack-kube-prom-kubelet                   ClusterIP      None             <none>                                                                        10250/TCP,10255/TCP,4194/TCP   2d19h

NAME                                  DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR              AGE
daemonset.apps/aws-node               2         2         2       2            2           <none>                     2d22h
daemonset.apps/ebs-csi-node           2         2         2       2            2           kubernetes.io/os=linux     2d21h
daemonset.apps/ebs-csi-node-windows   0         0         0       0            0           kubernetes.io/os=windows   2d21h
daemonset.apps/kube-proxy             2         2         2       2            2           <none>                     2d22h
daemonset.apps/s3-csi-node            2         2         2       2            2           kubernetes.io/os=linux     2d21h

NAME                                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/aws-load-balancer-controller   2/2     2            2           2d21h
deployment.apps/coredns                        2/2     2            2           2d22h
deployment.apps/ebs-csi-controller             2/2     2            2           2d21h
deployment.apps/external-dns                   1/1     1            1           41h
deployment.apps/kube-ops-view                  1/1     1            1           2d21h

NAME                                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/aws-load-balancer-controller-79574fb866   2         2         2       2d21h
replicaset.apps/coredns-56dfff779f                        2         2         2       2d22h
replicaset.apps/ebs-csi-controller-66ccf4755d             2         2         2       2d21h
replicaset.apps/external-dns-7cfc59cdbf                   1         1         1       41h
replicaset.apps/kube-ops-view-9cc4bf44c                   1         1         1       2d21h
---------------------------------------------------------------------------------
$ kubectl get all -n argocd

NAME                                                    READY   STATUS    RESTARTS   AGE
pod/argocd-application-controller-0                     1/1     Running   0          2d21h
pod/argocd-applicationset-controller-584f68b9d7-lc94c   1/1     Running   0          2d21h
pod/argocd-dex-server-8577d9498b-8zrd8                  1/1     Running   0          2d21h
pod/argocd-notifications-controller-564dcb4995-rm5cs    1/1     Running   0          2d21h
pod/argocd-redis-66d9777b78-x7mk4                       1/1     Running   0          2d21h
pod/argocd-repo-server-58c94b5cbf-8s7v8                 1/1     Running   0          2d21h
pod/argocd-server-b8bd4f4b5-nfmcf                       1/1     Running   0          2d21h

NAME                                              TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/argocd-applicationset-controller          ClusterIP   172.20.189.66    <none>        7000/TCP,8080/TCP            2d21h
service/argocd-dex-server                         ClusterIP   172.20.5.11      <none>        5556/TCP,5557/TCP,5558/TCP   2d21h
service/argocd-metrics                            ClusterIP   172.20.116.128   <none>        8082/TCP                     2d21h
service/argocd-notifications-controller-metrics   ClusterIP   172.20.243.0     <none>        9001/TCP                     2d21h
service/argocd-redis                              ClusterIP   172.20.53.96     <none>        6379/TCP                     2d21h
service/argocd-repo-server                        ClusterIP   172.20.238.149   <none>        8081/TCP,8084/TCP            2d21h
service/argocd-server                             ClusterIP   172.20.212.138   <none>        80/TCP,443/TCP               2d21h
service/argocd-server-metrics                     ClusterIP   172.20.31.63     <none>        8083/TCP                     2d21h

NAME                                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/argocd-applicationset-controller   1/1     1            1           2d21h
deployment.apps/argocd-dex-server                  1/1     1            1           2d21h
deployment.apps/argocd-notifications-controller    1/1     1            1           2d21h
deployment.apps/argocd-redis                       1/1     1            1           2d21h
deployment.apps/argocd-repo-server                 1/1     1            1           2d21h
deployment.apps/argocd-server                      1/1     1            1           2d21h

NAME                                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/argocd-applicationset-controller-584f68b9d7   1         1         1       2d21h
replicaset.apps/argocd-dex-server-8577d9498b                  1         1         1       2d21h
replicaset.apps/argocd-notifications-controller-564dcb4995    1         1         1       2d21h
replicaset.apps/argocd-redis-66d9777b78                       1         1         1       2d21h
replicaset.apps/argocd-repo-server-58c94b5cbf                 1         1         1       2d21h
replicaset.apps/argocd-server-b8bd4f4b5                       1         1         1       2d21h

NAME                                             READY   AGE
statefulset.apps/argocd-application-controller   1/1     2d21h
@Leeeuijooo Leeeuijooo added the bug Something isn't working label Mar 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Leeeuijooo