Pod metric not being read

[vobrien-axway]

I have updated my config with the following kuberenetesMetricSpec:

kubernetesMetricSpecs:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
  - type: Pods
    pods:
      metric:
        name: kube_pod_info
      target:
        type: AverageValue
requireKubernetesMetrics: true
runMode: "per-pod"

Within the returned data I see both the memory and cpu resource kubernetesMetrics fields, but that is it, the 3rd Pods field doesn't show up at all. I have tried both with runMode set to: per-pod and per-resource.

If I configure a HPA with the same values, all show up and are providing data.

I assume I just have something misconfigured but can't figure out what and I don't see any errors.

Any suggestions

[jthomperoo]

I've modified the example k8s-metrics-cpu to use your metrics specs, and I'm seeing the same behaviour. I'll look into this a bit more, this could be a bug in the CPA.

[vobrien-axway]

I actually managed to get passed it. I removed the Resources entirely and Pods started working, so then added them back in, with the Pods type first and Resources after and in continued working. Seems like if anything comes after Resource it gets ignored?

[jthomperoo]

That's strange! Thanks for narrowing that down, seems to definitely be some weird behaviour here - I'll have a poke around. Thanks!

[jthomperoo]

There seems to be a bit of a problem with the underlying k8shorizmetrics library the CPA uses for gathering metrics. If you are fetching multiple metrics and some but not all of them fail to gather the library silently swallows the errors for the failed metric gathering (code here).

I think this was masking some other errors that the K8s metrics API would have reported, so I've created a ticket on that library to get this behaviour changed: jthomperoo/k8shorizmetrics#10

I'll get this updated in the CPA soon too so errors are not swallowed and are logged out properly.

Thanks for spotting this!

Update:

This should no longer be an issue in v2.11.0 and above, errors are now properly logged due to #195.