CI: ci-external-workload: Cilium API handler panicked

[borkmann]

CI failure

Seen in one of the agent logs attached to the zip in here:

2022-04-13T08:05:39.897310085Z level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
2022-04-13T08:07:15.761674513Z level=info msg="regenerating all endpoints" reason="one or more identities created or deleted" subsys=endpoint-manager
2022-04-13T08:08:21.258858526Z level=warning msg="Cilium API handler panicked" client=@ method=GET panic_message="json: unsupported value: NaN" subsys=api url=/v1/metrics/
2022-04-13T08:08:21.259518780Z 2022/04/13 08:08:21 http: superfluous response.WriteHeader call from github.com/cilium/cilium/pkg/api.(*APIPanicHandler).ServeHTTP.func1 (apipanic.go:36)

https://github.com/cilium/cilium/actions/runs/2158036465
cilium-sysdump-out.zip (23).zip

[aanm]

Marking as a release blocker because the cilium metrics list file is empty and this file is important for debugging purposes.

[ti-mo]

I'm not sure if this is a flake, it's breaking #19159 CI for me on every run. It's causing so much noise that I can't see actual failures anymore.

This took a while for me to narrow this down to cilium metrics list being the culprit. APIPanicHandler recovers the json panic, but WriteHeader() at apipanic.go:36 seems to cause another panic.

I think 6c5e2d6 is the culprit. If I dump var metrics in getMetrics.Handle(), I see these NaNs popping up (when running cilium metrics list):

{map[] go_gc_pauses_seconds_total NaN}
{map[] go_sched_latencies_seconds NaN}

As suggested in golang/go#25721 (comment), this might require implementing a custom codec for models.Metric.

[chancez]

I don't believe 6c5e2d6 is the cause, but the recent bump to prometheus/client_golang is. See: d8e3f28.
Newer versions of prometheus/client_golang introduced new metrics, and they seem to have some problems.

This bug has been reported upstream, and it seems they reverted adding the new metrics:
prometheus/client_golang#995
prometheus/client_golang#1033

[chancez]

I suggest reverting the update to prometheus/client_golang so we're at 1.11 until a new release is made to fix the new metrics introduced in 1.12.

[ti-mo]

Fix proposed at: #19496.

[ti-mo]

This issue was reintroduced by f321a6a, which upgraded client_golang to 1.12.1 again. 1.12.2 is out, which supposedly fixes this.