Issue with GKE, Grafana, CertManager, SonarQube latest version and SHA tag and old metric

[AleksandarMis]

Hi,

First of all, thank you very much for the very helpful service version checker :) !

We have k8s running on GCP and the latest version-checker version:
Version-checker: quay.io/jetstack/version-checker:v0.3.3

In our config for the deployment of version-checker we added:
a flag:
"--test-all-containers=true"
annotations:
enable.version-checker.io/version-checker: "true"
use-sha.version-checker.io/version-checker: "false"

For the most of components and pods it works good and with exactly versions, but with some of them we have issues:

1. GKE
   
   As you can see, our current gke version is v.1.26.5-gke.2700 and the latest version found by version-checker is v1.18.6
   We have checked the logs and found following error:

"textPayload": "time="2024-01-04T15:52:04Z" level=error msg="error syncing 'kube-proxy-gke-staging-app-pool-a-xxxxx/kube-system': failed to sync pod kube-proxy-gke-staging-app-pool-a-xxxxxxx/kube-system: failed to check container image \"kube-proxy\": failed to get tags from remote registry for \"gke.gcr.io/kube-proxy-amd64\": failed to get docker image: Get \"https://gcr.io/v2/google-containers/kube-proxy-amd64/tags/list\\\": context canceled, requeuing" module=controller",

Actually https://gcr.io/v2/google-containers/kube-proxy-amd64/tags/list hasn't been updated for a long time because they switched to https://gke.gcr.io/v2/kube-proxy-amd64/tags/list and new images are published there.

The latest version of gke: v1.29.0-gke.1324000

P.S. kube-proxy is deployed as a static Pod for nodes.

2. Grafana
   The same issue as described here False positives for strange tag versions #138

For the image grafana/grafana version-checker says the latest version is 9799770991 and the actual latest tag currently would be 10.3.1.

3. CertManager

For the image quay.io/jetstack/cert-manager-controller version-checker says the latest version is 608111629 and the actual latest tag currently would be 1.13.3

4.SonarQube
For the image sonarqube version-checker says the latest version is 7.10 and the actual latest tag currently would be 10.3

The reason behin that is that we use image from our jfrog where is authentication required and it fails with error UNAUTHORIZED.

I tried overwriting the URL in sonarqube deployment to check for the latest image in https://hub.docker.com/_/sonarqube , but it doesn't help and I can't find any error after overwriting and redeploying.

5. SHA tag
   Many images with sha tag are sent to prometheus by version-checker and are then recognized as not being the latest (as in example below), because current_version is the same but only without or different sha tag.
   I wanted to disable sha tag and test on version-checker itself, with use-sha.version-checker.io/version-checker: "false" and version-checker latest images with sha are still shown and recognized as "Is latest" on some stages and not on others NOT, even though they are EXACTLY THE SAME VERSIONS:

Since we have a lot of components and plan maintanance appendix to the filter "Is latest=NO", then we are shown wrong and we have to check manually, so it would be great if we could fix that too.

6. Version-checker send old metrics from old containers as current
   We have also following issue, that version-checker sends old metrics ( from containers that are not existing anymore) to prometheus.
   One example:
   

And we have only one runner with version 16.7.0 and the old one was terminated 10 days ago.

One more example:

Also just one pod with version 3.9.0 and the metrics that are sent to prometheus are:

http://localhost:8080/metrics

_version_checker_is_latest_version{container="grafana-renderer",container_type="container",current_version="3.8.4",image="grafana/grafana-image-renderer",latest_version="3.9.0",namespace="grafana",pod="grafana-renderer-859948fb9f-wrbzb"} 0

version_checker_is_latest_version{container="grafana-renderer",container_type="container",current_version="3.9.0@sha256:656ca4dddc020f067239428e2a15bc7100d8ce4918db1618b45d53d0c8c4d273",image="grafana/grafana-image-renderer",latest_version="3.9.0@sha256:a1e0c69aaa5c1fe106c89ba4c5569563d8b2ac0b04e0f121b12b5c2a5b4c3f94",namespace="grafana",pod="grafana-renderer-545676cb7d-hd8lm"} 1_

We have scrape_interval for serviceMonitor 30s.

` -job_name: serviceMonitor/version-checker/version-checker/0
honor_timestamps: true
scrape_interval: 30s
scrape_timeout: 10s
metrics_path: /metrics
scheme: http
follow_redirects: true
enable_http2: true
relabel_configs:

• source_labels: [job]
  separator: ;
  regex: (.*)
  target_label: __tmp_prometheus_job_name
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_label_app, __meta_kubernetes_service_labelpresent_app]
  separator: ;
  regex: (version-checker);true
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_port_name]
  separator: ;
  regex: web
  replacement: $1
  action: keep
• source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
  separator: ;
  regex: Node;(.*)
  target_label: node
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_endpoint_address_target_kind, __meta_kubernetes_endpoint_address_target_name]
  separator: ;
  regex: Pod;(.*)
  target_label: pod
  replacement: ${1}
  action: replace
• source_labels: [__meta_kubernetes_namespace]
  separator: ;
  regex: (.*)
  target_label: namespace
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: service
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_name]
  separator: ;
  regex: (.*)
  target_label: pod
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_container_name]
  separator: ;
  regex: (.*)
  target_label: container
  replacement: $1
  action: replace
• source_labels: [__meta_kubernetes_pod_phase]
  separator: ;
  regex: (Failed|Succeeded)
  replacement: $1
  action: drop
• source_labels: [__meta_kubernetes_service_name]
  separator: ;
  regex: (.*)
  target_label: job
  replacement: ${1}
  action: replace
• separator: ;
  regex: (.*)
  target_label: endpoint
  replacement: web
  action: replace
• source_labels: [address]
  separator: ;
  regex: (.*)
  modulus: 1
  target_label: __tmp_hash
  replacement: $1
  action: hashmod
• source_labels: [__tmp_hash]
  separator: ;
  regex: "0"
  replacement: $1
  action: keep
  kubernetes_sd_configs:
• role: endpoints
  kubeconfig_file: ""
  follow_redirects: true
  enable_http2: true
  namespaces:
  own_namespace: false
  names:
  • version-checker`

Is there any config to add/change to send only the metric from latest scan?

The only solution currently is to redeploy the version checker after each component version upgrade and then will only send the latest version and not the old one.

Could you please help?

Thanks in advance!

BR Aleks

[hawksight]

@AleksandarMis thank you for raising such a detailed issue for us. I need a bit of time to digest all the issues presented here and determine if we already have similar or duplicated issues open, such as #138 as you already mentioned.

Just wanted to say we've seen your issue and I plan to take a look in the next few weeks.