Add Prometheus metrics validation

[doniacld]

Motivation

Add the possibility to verify metrics from the Cilium agent pod and validate it has the expected behaviour before and after an action.

Changes

• commit 1 Add a k8s dialer to be able to port forward /metrics endpoint outside of the pod
• commit 2 Add the logic to call /metrics endpoint and parse the metrics to retrieve them into a nice usable open metrics format
• commit 3 policy.go is a pretty long file now, let's move Result with its associated methods in its own file.
• commit 4 Make the call to collect metrics in the action
• commit 5 Validate the metrics by defining a new type metricsCompareFunc which will compare metrics in a generic way.
• commit 6 Add a new scenario very basic with a simple metric to verify the feature is working

Usage

Implement the operation you want to check on your metrics on Result:

func (r Result) ExpectMetricsIncrease(source MetricsSource, metrics ...string) Result {

Then use it directly in your expectations by passing a MetricSource type (cilium-agent, cilium-operator...) and the name of the metric you want to check:

return check.ResultOK.ExpectMetricsIncrease(ct.CiliumAgentMetrics(), "cilium_forward_count_total")

[gandro]

Awesome work, thanks a lot! I've left some feedback regarding the open TODOs.

[gandro]

Awesome! High-level this looks good to me. I did leave some minor feedback that should be addressed before we can merge. Thanks for working on this!

[gandro]

Awesome, overall looks good to me! A couple minor nits remaining

[gandro]

Looks good to me! Two minor (non-blocking) nits remaining

[brlbil]

The CI part is 👍 Awesome work, thanks a lot!

[tklauser]

Nice work!