Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add prometheus metrics emitter #187

Merged
merged 5 commits into from
Jul 24, 2023
Merged

Add prometheus metrics emitter #187

merged 5 commits into from
Jul 24, 2023

Conversation

bluekeyes
Copy link
Member

@bluekeyes bluekeyes commented Jul 3, 2023

The core of this package is a prometheus.Collector implementation that exposes the metrics of an rcrowley/go-metrics registry. It also provides a convenience function for creating a collector, registering it with a Prometheus registry, and exposing it on a HTTP endpoint. If you need to customize any of the Prometheus options, you can use the Collector directly.

I ended up writing my own emitter because the existing exporters I found had some limitations:

  • Don't understand our custom tag/label format
  • Report at fixed intervals instead of implementing Collector
  • Export histograms incorrectly

Because this is a new emitter, I tried to stay closer to Prometheus conventions rather than matching the Datadog emitter. Specifically:

  • Some metrics have different names (e.g "mean" instead of "avg") to match the go-metrics terminology
  • Timers are always exported in fractional seconds
  • Medians are reported as the 0.5 quantile instead of as a separate named metric

But I could be convinced it's better to match the Datadog emitter for easier migration.

The core of this package is a prometheus.Collector implementation that
exposes the metrics of an rcrowley/go-metrics registry. It also provides
a convenience function for creating a collector, registering it with a
Prometheus registry, and exposing it on a HTTP endpoint. If you need to
customize any of the Prometheus options, you can use the Collector
directly.

I ended up writing my own emitter because the existing exporters I found
had some limitations:

* Don't understand our custom tag/label format
* Report at fixed intervals intead of implementing Collector
* Export histograms incorrectly
Copy link
Member

@asvoboda asvoboda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems pretty straightforward, thanks for implementing it! Do you think we should also add some basic functionality tests as well?

@bluekeyes
Copy link
Member Author

bluekeyes commented Jul 5, 2023

Yeah, some basic tests would be good. I started writing some and discovered an issue with the Prometheus testing library, so I'm trying to decide how to best work around that.

Per https://prometheus.io/docs/instrumenting/writing_exporters/, remove
less useful statistics that can be computed in Prometheus, add simple
help text to indicate the original metric type, add "seconds" to the
names of timer metrics, and improve the santize function to avoid long
runs of underscores.
@bluekeyes
Copy link
Member Author

bluekeyes commented Jul 20, 2023

Updated to add tests. I also made some other changes:

  • https://prometheus.io/docs/instrumenting/writing_exporters/ suggests not exporting statistics that you can calculate in Prometheus and specifically uses Dropwizard as an example, so I've removed most of those values.
  • I added help text giving the original metric type. This was mostly to make the tests work, but might be useful.
  • I added seconds to the name of all timer metrics
  • The sanitize function replaces runs of invalid characters with a single underscore

Copy link
Member

@asvoboda asvoboda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bluekeyes bluekeyes merged commit 026c830 into develop Jul 24, 2023
3 checks passed
@bluekeyes bluekeyes deleted the bkeyes/prometheus-export branch July 24, 2023 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants