Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dashboard helm chart 7.3.2 no graphics #8997

Closed
Alezzzander opened this issue May 2, 2024 · 14 comments
Closed

Dashboard helm chart 7.3.2 no graphics #8997

Alezzzander opened this issue May 2, 2024 · 14 comments
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@Alezzzander
Copy link

What happened?

Moved from 2.7.0 dashboard version to the new dashboard-web 1.3.0.
Metrics-server and metrics-scrapper are enabled and installed from dashboard helm chart. No errors/warns in logs. Kubectl top nodes works good but all graphics are gone. Don't know how to troubleshoot it .Any suggestions?

What did you expect to happen?

Expecting to see graphics like in kubernetes-dashboard 2.7.0

How can we reproduce it (as minimally and precisely as possible)?

1.Install EKS 1.29
2.Install dashboard helm chart with metrics-scrapper and metrics-server enabled.

Anything else we need to know?

No response

What browsers are you seeing the problem on?

Chrome

Kubernetes Dashboard version

1.3.0

Kubernetes version

1.29

Dev environment

No response
@Alezzzander Alezzzander added the kind/bug Categorizes issue or PR as related to a bug. label May 2, 2024
@floreks
Copy link
Member

floreks commented May 2, 2024

You need to provide logs from all dashboard containers first.

@Alezzzander
Copy link
Author

During log collection,i found this line in dashboard-api container:

E0502 11:07:49.844195       1 manager.go:96] Metric client health check failed: the server is currently unable to handle the request (get services kubernetes-dashboard-metrics-scraper). Retrying in 30 seconds.

Can it be the reason?

dashboard-api has these args:

  containers:
    - args:
        - '--namespace=kube-system'
        - '--metrics-scraper-service-name=kubernetes-dashboard-metrics-scraper'
      env:
        - name: CSRF_KEY
          valueFrom:
            secretKeyRef:
              key: private.key
              name: kubernetes-dashboard-csrf
      image: docker.io/kubernetesui/dashboard-api:1.5.0

dashboard-scraper logs are filled with probes:

10.150.23.46 - - [02/May/2024:11:05:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:05:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:06:02.711109       1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:06:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:06:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:07:02.747414       1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:07:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:07:58 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
I0502 11:08:02.701078       1 main.go:145] Database updated: 12 nodes, 310 pods
10.150.23.46 - - [02/May/2024:11:08:08 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:18 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:28 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:38 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"
10.150.23.46 - - [02/May/2024:11:08:48 +0000] "GET / HTTP/1.1" 200 6 "" "kube-probe/1.29+"

metrics-server logs:


I0502 10:12:39.303482       1 serving.go:374] Generated self-signed cert (/tmp/apiserver.crt, /tmp/apiserver.key)
I0502 10:12:39.861123       1 handler.go:275] Adding GroupVersion metrics.k8s.io v1beta1 to ResourceManager
I0502 10:12:39.979541       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0502 10:12:39.979591       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0502 10:12:39.979654       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0502 10:12:39.979740       1 shared_informer.go:311] Waiting for caches to sync for RequestHeaderAuthRequestController
I0502 10:12:39.979763       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0502 10:12:39.979852       1 shared_informer.go:311] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0502 10:12:39.980179       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tmp/apiserver.crt::/tmp/apiserver.key"
I0502 10:12:39.981162       1 secure_serving.go:213] Serving securely on [::]:10250
I0502 10:12:39.981201       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0502 10:12:40.080042       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0502 10:12:40.080062       1 shared_informer.go:318] Caches are synced for RequestHeaderAuthRequestController
I0502 10:12:40.080152       1 shared_informer.go:318] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file

Metrics-server args:

 --secure-port=10250 
 --cert-dir=/tmp 
 --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname 
 --kubelet-use-node-status-port 
 --metric-resolution=15s 
 --kubelet-insecure-tls

@floreks
Copy link
Member

floreks commented May 2, 2024

Ye, for some reason it can't access metrics scraper. Isn't there more detailed error message in the api logs?

@Alezzzander
Copy link
Author

Only these :
image

@floreks
Copy link
Member

floreks commented May 2, 2024

Check the beginning of logs when the app is being initialized.

@Alezzzander
Copy link
Author

Alezzzander commented May 2, 2024 •
edited 

I0502 11:19:08.330608       1 main.go:40] "Starting Kubernetes Dashboard API" version="1.5.0"
I0502 11:19:08.331182       1 init.go:47] Using in-cluster config
I0502 11:19:08.364971       1 main.go:118] "Successful initial request to the apiserver" version="v1.29.3-eks-adc7111"
I0502 11:19:08.365756       1 client.go:268] Creating in-cluster Sidecar client
I0502 11:19:08.381661       1 main.go:95] "Listening and serving on" address="0.0.0.0:8000"
2024/05/02 11:19:08 Getting list of all cron jobs in the cluster
2024/05/02 11:19:09 Getting list of all replica sets in the cluster
2024/05/02 11:19:09 Getting list of all replication controllers in the cluster
2024/05/02 11:19:09 Getting list of ingress classes in the cluster
2024/05/02 11:19:09 Getting list of all services in the cluster
2024/05/02 11:19:09 Getting list of all stateful sets in the cluster
2024/05/02 11:19:09 Getting list of all pods in the cluster
2024/05/02 11:19:09 Getting list of all deployments in the cluster
2024/05/02 11:19:09 Getting list of all jobs in the cluster
2024/05/02 11:19:09 Getting list config maps in the namespace 
2024/05/02 11:19:09 Getting list of secrets in &{[]} namespace
2024/05/02 11:19:09 Getting list persistent volumes
2024/05/02 11:19:09 Getting list of RBAC roles
2024/05/02 11:19:09 Getting list of storage classes in the cluster
2024/05/02 11:19:09 Getting list of all roleBindings in the cluster
2024/05/02 11:19:09 Getting list of all roles in the cluster
2024/05/02 11:19:09 Getting list of namespaces
2024/05/02 11:19:09 Getting list of all clusterRoleBindings in the cluster
2024/05/02 11:19:09 Getting list of all deployments in the cluster
2024/05/02 11:19:13 Getting list persistent volumes
2024/05/02 11:19:18 Getting pod metrics
2024/05/02 11:19:39 Getting list of all deployments in the cluster
2024/05/02 11:19:54 Getting list of all deployments in the cluster
2024/05/02 11:19:58 Getting list of namespaces
2024/05/02 11:19:58 Getting list of all cron jobs in the cluster
2024/05/02 11:19:58 Getting list of all pods in the cluster
2024/05/02 11:19:58 Getting list of all replication controllers in the cluster
2024/05/02 11:19:58 Getting list of all deployments in the cluster
2024/05/02 11:19:58 Getting list of all jobs in the cluster
2024/05/02 11:19:58 Getting list of all replica sets in the cluster
2024/05/02 11:19:58 Getting list of all stateful sets in the cluster
2024/05/02 11:19:58 Getting pod metrics
2024/05/02 11:19:59 Getting list of namespaces
2024/05/02 11:19:59 Getting list of all cron jobs in the cluster
2024/05/02 11:19:59 Getting list of all deployments in the cluster
2024/05/02 11:19:59 Getting list of all deployments in the cluster
2024/05/02 11:19:59 Getting list of all jobs in the cluster
2024/05/02 11:19:59 Getting list of all replication controllers in the cluster
2024/05/02 11:19:59 Getting list of all replica sets in the cluster
2024/05/02 11:19:59 Getting list of all pods in the cluster
2024/05/02 11:19:59 Getting list of all stateful sets in the cluster
2024/05/02 11:19:59 Getting pod metrics
2024/05/02 11:20:01 Getting list of all deployments in the cluster
2024/05/02 11:20:02 Getting list of all deployments in the cluster
2024/05/02 11:20:03 Getting list of all jobs in the cluster
2024/05/02 11:20:04 Getting list of all pods in the cluster
2024/05/02 11:20:04 Getting list of all jobs in the cluster
2024/05/02 11:20:04 Getting list of all deployments in the cluster
2024/05/02 11:20:04 Getting list of all cron jobs in the cluster
2024/05/02 11:20:04 Getting list of namespaces
2024/05/02 11:20:04 Getting list of all stateful sets in the cluster
2024/05/02 11:20:04 Getting list of all replication controllers in the cluster
2024/05/02 11:20:04 Getting list of all replica sets in the cluster
2024/05/02 11:20:04 Getting pod metrics
2024/05/02 11:20:04 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all deployments in the cluster
2024/05/02 11:20:06 Getting list of all cron jobs in the cluster
2024/05/02 11:20:06 Getting list of namespaces
2024/05/02 11:20:06 Getting list of all replication controllers in the cluster
2024/05/02 11:20:06 Getting list of all replica sets in the cluster
2024/05/02 11:20:06 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all stateful sets in the cluster
2024/05/02 11:20:06 Getting pod metrics
2024/05/02 11:20:06 Getting list of all pods in the cluster
2024/05/02 11:20:06 Getting list of all jobs in the cluster
2024/05/02 11:20:07 Getting list of all deployments in the cluster
2024/05/02 11:20:07 Getting pod metrics
2024/05/02 11:20:09 Getting list of all deployments in the cluster
2024/05/02 11:20:10 Getting pod metrics
2024/05/02 11:20:11 Getting list of namespaces
2024/05/02 11:20:11 Getting list of all pods in the cluster
2024/05/02 11:20:11 Getting pod metrics
2024/05/02 11:20:12 Getting list of all deployments in the cluster
2024/05/02 11:20:13 Getting list of all replica sets in the cluster
2024/05/02 11:20:14 Getting list of namespaces
2024/05/02 11:20:15 Getting list of all replica sets in the cluster
2024/05/02 11:20:17 Getting list of all deployments in the cluster
2024/05/02 11:20:17 Getting list of all deployments in the cluster
2024/05/02 11:20:18 Getting list of namespaces
2024/05/02 11:20:20 Getting details of kubernetes-dashboard-metrics-server deployment in kube-system namespace
2024/05/02 11:20:22 Getting list of all deployments in the cluster
2024/05/02 11:20:24 Getting list of namespaces
2024/05/02 11:20:26 Getting details of kubernetes-dashboard-metrics-server deployment in kube-system namespace
2024/05/02 11:20:26 Getting replication controller kubernetes-dashboard-metrics-server-7c54c778c6 pods in namespace kube-system
2024/05/02 11:20:26 Getting details of kubernetes-dashboard-metrics-server-7c54c778c6 service in kube-system namespace
2024/05/02 11:20:27 Getting list of all deployments in the cluster
2024/05/02 11:20:27 Getting pod metrics
2024/05/02 11:20:28 Getting list of namespaces
2024/05/02 11:20:31 Getting replication controller kubernetes-dashboard-metrics-server-7c54c778c6 pods in namespace kube-system
2024/05/02 11:20:31 Getting details of kubernetes-dashboard-metrics-server-7c54c778c6 service in kube-system namespace
2024/05/02 11:20:31 Getting list of all deployments in the cluster
2024/05/02 11:20:32 Getting pod metrics
2024/05/02 11:20:33 Getting list of namespaces
2024/05/02 11:20:36 Getting list of all deployments in the cluster
2024/05/02 11:20:37 Getting list of all deployments in the cluster
2024/05/02 11:20:38 Getting list of all deployments in the cluster
2024/05/02 11:20:38 Getting list of namespaces
2024/05/02 11:20:43 Getting list of namespaces
2024/05/02 11:20:45 Getting pod metrics
2024/05/02 11:20:48 Getting list of namespaces
2024/05/02 11:20:53 Getting list of namespaces
2024/05/02 11:20:58 Getting list of namespaces
2024/05/02 11:21:03 Getting list of namespaces
2024/05/02 11:21:08 Getting list of namespaces
2024/05/02 11:21:13 Getting list of namespaces
E0502 11:21:18.808202       1 manager.go:96] Metric client health check failed: the server is currently unable to handle the request (get services kubernetes-dashboard-metrics-scraper). Retrying in 30 seconds.
2024/05/02 11:21:18 Getting list of namespaces
2024/05/02 11:21:23 Getting list of namespaces
2024/05/02 11:21:28 Getting list of namespaces

Here is the log from the beginning

@viram99
Copy link

viram99 commented May 7, 2024

I have used the same configuration and observed similar logs.
I'm encountering the same issue where no graphics are displayed on the Kubernetes dashboard.

@floreks
Copy link
Member

floreks commented May 7, 2024

You could check if this helps: #8835 (comment)

@viram99
Copy link

viram99 commented May 7, 2024 •
edited

I have tried adding below args but it still does not work for me.
args:

  • --metrics-provider=none
  • --sidecar-host=kubernetes-dashboard-metrics-scraper.kubernetes-dashboard

This is the API response for request path /api/v1/pod/ that I have received. any insights to why cumulativeMetrics and metrics field are set to empty and null respectively
{ "listMeta": { "totalItems": 2 }, "cumulativeMetrics": [], "status": { "running": 2, "pending": 0, "failed": 0, "succeeded": 0, "terminating": 0 }, "pods": [{...other pod fields, "metrics":"null" }], "errors": [] }

@floreks
Copy link
Member

floreks commented May 7, 2024

If you set metrics provider to none you are effectively disabling metrics :). Do not override it or use 'sidecar' as value.

@sigfigsteve
Copy link

It needs a protocol and port for the sidecar-host
--sidecar-host=http://kubernetes-dashboard-metrics-scraper.kubernetes-dashboard:8000

@floreks
Copy link
Member

floreks commented May 9, 2024

@viram99 any progress here?

@floreks
Copy link
Member

floreks commented May 27, 2024

/close

@k8s-ci-robot
Copy link
Contributor

@floreks: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@floreks @k8s-ci-robot @Alezzzander @sigfigsteve @viram99