Create bare exporter for allowing metrics to be arbitrarily collected.

[tobz]

Over time, metrics-exporter-prometheus has received the lion's share of features and updates: clearing idle metrics, handling aggregation of histogram data, and so on.

While most people can use a Prometheus exporter to integrate with their existing observability pipelines, some users have requested the ability to access the metric data stored by metrics-exporter-prometheus directly from their applications, but with all transformations, aggregations, etc, still present.

One PR (#236) sought to add this capability directly to metrics-exporter-prometheus but this sort of flies in the face of the entire point of a Prometheus-specific exporter. However, with a small bit of work, we could likely abstract this functionality from metrics-exporter-prometheus and create a new exporter that only provides the ability to access the metrics after processing idle metrics, aggregation, and so on.

Additionally, this would make it easier to design other exporters since most of the common necessities -- don't send idle metrics, send histograms in whatever the most granular form accepted is, etc -- could be abstracted and then refactored slightly to become more like stackable middleware than specific/hard-coded bolt-ons to certain exporters.

[dezyh]

I'm going to take a shot at this.

Personally, my motivation for this is just being able to register multiple exporters (for example: prometheus + interval logger). I also would like #243 which I think could maybe be solved/included in this too. I will consider the ability to include metrics-exporter-prometheus' features (transformations, aggregations, etc) in the design but I probably won't port them to this solution right away.

Admittedly, I only have a rough idea of how to approach this. So I'm thinking it might be best to hack together something first to better understand the internals, then check back in to get some feedback/review from the maintainers.

I will update when I make progress on this.

[tobz]

👋🏻

One thing I would say is to be cognizant of the boundary between recording and transforming. The recording side is more or less fixed as it stands with Recorder, and we don't want to do anything that requires changing it. Everything that we want to be able to do more generically should come on the "transforming" side, which is generally once the metric has made it to storage (so this would be, for example, when a metric exists in the Registry<K, S> for the Prometheus exporter).

Since Registry<K, S> is the common building block for most real exporter implementations, it could make sense to think about coming up with a corresponding trait that represents the other side of Recorder, which is actually reading/writing to storage. You could imagine some trait called, I don't know, Backend<K, S>, where K and S come from Registry<K, S>. The trait would have read and write methods, similar to what Registry<K, S> has now but perhaps generalized a bit. If done right, what we could then do is start to write middleware that does this additional logic that we want to share across exporter implementations.

As an example, we might wrap an "remove idle metrics" middleware around Registry<K, S>, where both of those things would implement this proposed trait. The read methods for the idle metrics middleware would do its interrogation of the metrics returned by the registry, and if it found idle metrics, it would call the underlying/wrapped backend and delete those metrics before returning the non-idle ones.

There's plenty of little nits and things to design around (like needing/wanting mutability in the exterior mutability) but I think something along these lines is probably the way forward for building more composable exporters and thus allowing more extensibility for existing exporters and more easily authoring new exporters as well.