Fix metrics labeling with only directed source/destination context op…

[marqc]

…tions set.

Fixes: #27717

Please ensure your pull request adheres to the following guidelines:

• All code is covered by unit and/or runtime tests where feasible.
• All commits contain a well written commit description including a title,
  description and a Fixes: #XXX line if the commit addresses a particular
  GitHub issue.
• All commits are signed off. See the section Developer’s Certificate of Origin
• Provide a title or release-note blurb suitable for the release notes.

In case only directed Source/Destination Ingress/Egress context options were specified, metrics were incorrectly labeled. this change ensures that correct labels are included when only directed options are specified in metrics configuration.

Fixes: #27717

Fix hubble metric labeling when only directed Source/Destination Ingress/Egress options are specified.

[gandro]

/test

[gandro]

/test

Edit: ci-ipsec-upgrade hit #27827, clearly unrelated to the PR since it doesn't affect the datapath or IPSec

[gandro]

Not sure why Travis was not started. I don't see any failure or pending run in the dashboard https://app.travis-ci.com/github/cilium/cilium/pull_requests

[gandro]

I'll close and re-open the PR to hopefully re-trigger Travis. This will invalidate the tests (which are all green at the momen ) unfortunately, but I don't think there is any other way.

[gandro]

Travis is refusing to acknowledge the existence of this PR. Not sure what else we can do.

I'll mark it ready to merge, since all other CI is green and the affected tests here seem to have passed in IntegrationTests

[matmerr]

@gandro not sure if ready to merge yet, I built this PR and ran some manual validation from the scenario in #27717 with the flow enabled,

node 1: agnhost-a
node 2: agnhost-b

k exec -it agnhost-a -- curl <agnhost-b ip>

and it seems that ingress is working, both egress for source and destination don't report any metrics. In the case of sourceEgressContext, source="" is just empty string, and case of destinationEgressContext, destination="" is empty string as well

hubble-metrics: flow:destinationIngressContext
Node 0
hubble_flows_processed_total{destination="kube-system/agnhost-a",protocol="TCP",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 4

Node 1
hubble_flows_processed_total{destination="kube-system/agnhost-b",protocol="TCP",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 4

hubble-metrics: flow:destinationEgressContext=pod
Node 0
empty response

Node 1
empty response

hubble-metrics: flow:sourceEgressContext=pod
Node 0
empty response

Node 1
empty response

hubble-metrics: flow:sourceIngressContext=pod
Node 0
hubble_flows_processed_total{protocol="TCP",source="kube-system/agnhost-b",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 4

Node 1
hubble_flows_processed_total{protocol="TCP",source="kube-system/agnhost-a",subtype="to-endpoint",type="Trace",verdict="FORWARDED"} 4

[marqc]

@matmerr I verified your scenario with these code changes and everything looks fine.

test pods

nginx-77b4fdf86c-7wflz                1/1     Running     0          8s      10.80.7.153   node-A
nginx-77b4fdf86c-v2pfw                1/1     Running     0          8s      10.80.6.184   node-B

network call:

kubectl exec -it nginx-77b4fdf86c-7wflz -- curl 10.80.6.184

Case 1

cilium config:

hubble-metrics: flow:destinationEgressContext=pod;labelsContext=source_pod,destination_pod,traffic_direction

metrics on node-A

hubble_flows_processed_total{>>>>destination="default/nginx-77b4fdf86c-7wflz"<<<<,destination_pod="nginx-77b4fdf86c-7wflz",protocol="TCP",source_pod="nginx-77b4fdf86c-v2pfw",subtype="to-endpoint",traffic_direction="egress",type="Trace",verdict="FORWARDED"} 3
hubble_flows_processed_total{>>>>destination="default/nginx-77b4fdf86c-v2pfw"<<<<,destination_pod="nginx-77b4fdf86c-v2pfw",protocol="TCP",source_pod="nginx-77b4fdf86c-7wflz",subtype="to-stack",traffic_direction="egress",type="Trace",verdict="FORWARDED"} 5

metrics on the node-B

<none>

Case 2

cilium config:

hubble-metrics: flow:sourceIngressContext=pod;labelsContext=source_pod,destination_pod,traffic_direction

metrics on node-A

<none>

metrics on the node-B

hubble_flows_processed_total{destination_pod="nginx-77b4fdf86c-7wflz",protocol="TCP",>>>>source="default/nginx-77b4fdf86c-v2pfw"<<<<,source_pod="nginx-77b4fdf86c-v2pfw",subtype="to-stack",traffic_direction="ingress",type="Trace",verdict="FORWARDED"} 3
hubble_flows_processed_total{destination_pod="nginx-77b4fdf86c-v2pfw",protocol="TCP",>>>>source="default/nginx-77b4fdf86c-7wflz"<<<<,source_pod="nginx-77b4fdf86c-7wflz",subtype="to-endpoint",traffic_direction="ingress",type="Trace",verdict="FORWARDED"} 5

[anubhabMajumdar]

sourceEgressContext=

Hi @marqc , can you check sourceEgressContext scenario? I think @matmerr faced some issue with that particular context.

[matmerr]

fwiw I was also validating with just hubble-metrics: flow:sourceEgressContext=pod, no additional labelsContext

[marqc]

@matmerr @anubhabMajumdar

Case 3

cilium config:

hubble-metrics: flow:sourceEgressContext=pod;labelsContext=source_pod,destination_pod,traffic_direction

metrics on node-A

hubble_flows_processed_total{
  destination_pod="nginx-77b4fdf86c-7wflz",
  protocol="TCP",
  source="default/nginx-77b4fdf86c-v2pfw",
  source_pod="nginx-77b4fdf86c-v2pfw",
  subtype="to-endpoint",
  traffic_direction="egress",
  type="Trace",
  verdict="FORWARDED"
} 5
hubble_flows_processed_total{
  destination_pod="nginx-77b4fdf86c-v2pfw",
  protocol="TCP",
  source="default/nginx-77b4fdf86c-7wflz",
  source_pod="nginx-77b4fdf86c-7wflz",
  subtype="to-stack",
  traffic_direction="egress",
  type="Trace",
  verdict="FORWARDED"
} 7

metrics on the node-B

<none>

Case 3.1

cilium config:

hubble-metrics: flow:sourceEgressContext=pod

metrics on node-A

hubble_flows_processed_total{
  protocol="TCP",
  source="default/nginx-77b4fdf86c-7wflz",
  subtype="to-stack",
  type="Trace",
  verdict="FORWARDED"
} 7
hubble_flows_processed_total{
  protocol="TCP",
  source="default/nginx-77b4fdf86c-v2pfw",
  subtype="to-endpoint",
  type="Trace",
  verdict="FORWARDED"
} 5

metrics on the node-B

<none>

[matmerr]

UT's seemed to be working for egress, and tested in fresh cluster which seems to be working. Not worth investigating the first one for the time being, lgtm.