Mixed metrics when enable Open Telemetry sink for EG CP & DP at the same time

[shawnh2]

Description:

If we enable the Open Telemetry sink for EG control-plane and data-plane at the same time, by retrieving metrics from OTel-Collector, we will get mixed metrics from both CP & DP.

Do we need to separate these metrics?

• maybe by adding a label field for each metrcis to indicate which it comes from (DP or CP) could avoid this

/cc @envoyproxy/gateway-maintainers

Repro steps:

• CP follows https://gateway.envoyproxy.io/contributions/design/eg-metrics/#example
• DP follows https://gateway.envoyproxy.io/v1.0.1/tasks/observability/proxy-observability/#metrics

Environment:
latest

[zirain]

TBH, I'm not a fan of OpenTelemetry Metrics, is that anything like pod name or identity could be found?

[shawnh2]

the only difference is: metrics for DP shall have label prefix named envoy, like:

but nothing for CP.

[zirain]

EG should send something for identity

[shawnh2]

If is possible, we can add namespace label for all the metrics from CP.

xref: #3393 (comment)