Metrics/prometheus - lack of metadata/labels enrichment for process_metrics

[AlmogAlmado]

Hi,

I've been using the Prometheus push gateway for a while, and it has been excellent. However, I'm currently facing a minor issue with the process_metrics where it seems the meta information cant be enrichment, unlike the pure Benthos event. This becomes problematic when multiple pods are up, as I cannot distinguish between them(CPU,Memory etc)

Ideally, I just need to add the hostname as a label to resolve this issue. Am I missing something, or is there a workaround available?

If development is required, I'd be happy to contribute the necessary changes inside metrics_prometheus.go or process_collector.go.

Thanks,
Almog

[mihaitodor]

Hey @AlmogAlmado 👋

Ideally, I just need to add the hostname as a label to resolve this issue.

You can try to do that as described here. Bloblang has a hostname() function that you can use like so: root.hostname = hostname() and that should set the hostname label to the value reported by that function.

[AlmogAlmado]

Hi @mihaitodor ,

Thanks for the quick assistance as usual. I'm not quite sure what to do, and if it will work. I'm unable to adjust anything in process_metrics like process_cpu_seconds_total.

My setup example:
metrics:
mapping: |
root = "benthos_"+this.replace("-", "").replace(".", "")
meta pod_name = hostname()
meta instance = hostname()
meta monitor = "benthos"
prometheus:
add_process_metrics: true
add_go_metrics: false
push_url: *****
push_interval: 10s
push_job_name: test

[mihaitodor]

Hi @AlmogAlmado, sorry for the late reply. I was away on vacation and catching up now. It turns out that Benthos doesn't add custom label to process metrics and Go metrics, which I think can be addressed via an enhancement using the approach from this StackOverflow answer. Feel free to open an issue and ask for this enhancement linking this conversation. Not sure when it might get addressed though.

[AlmogAlmado]

Thank you, @mihaitodor, for your assistance.
I hope to create a pull request for this in the near future.

[AlmogAlmado]

@Jeffail
I'd appreciate your insights on the current state of process_metrics in the code. It seems like the adjustability is limited, especially for meta or root. Could you please share your point of view on this? Any guidance or advice would be incredibly helpful.

[AlmogAlmado]

@Jeffail

I'd appreciate hearing your point of view on this. Any insights you could share would be valuable. Thanks!

[AlmogAlmado]

@Jeffail
Would you mind share your POV?