Implement Prometheus statser

[tiedotguy]

This would allow for internal metrics to be monitored from prom.

This could be useful when there's a large number of ephemeral hosts, and other backends may not scale as well with the high cardinality of internal metrics across many hosts.

This is mostly stand-alone and doesn't interact with the more complicated aspects of how gostatsd handles things, as it's mostly just adding an http server + prom, so it's probably a good issue for someone looking to get across the code base.

This is explicitly not a prom backend, this is only for internal metrics, see #285 for that.

[0xYao]

This would allow for internal metrics to be monitored from prom.

This could be useful when there's a large number of ephemeral hosts, and other backends may not scale as well with the high cardinality of internal metrics across many hosts.

This is mostly stand-alone and doesn't interact with the more complicated aspects of how gostatsd handles things, as it's mostly just adding an http server + prom, so it's probably a good issue for someone looking to get across the code base.

This is explicitly not a prom backend, this is only for internal metrics, see #285 for that.

Hey, I am keen to give this issue a go. All good if I take it from here?

[tiedotguy]

That'd be great!

[0xYao]

Would pkg/web be the right place to make the changes, or is there a more suitable directory where I can add the HTTP server with Prometheus to?

[tiedotguy]

I'm not familiar with the Prom SDK in go, but my approach would be to create an appropriate PromStatser (which also includes the necessary Handler) in createStatser, then propagate that through to NewHttpServer, then if it's enabled (for that particular http server), attempt a type assertion to convert the interface to a PromStatser, and then attach the Handler if it's the right type.

If it's the wrong type, return an error - fast fail on startup is encouraged.

[0xYao]

When you said if it's enabled, do you mean if the flag for such as enablePrometheus is set to true? For example, in the NewHttpServer function in pkg/web/httpservers.go, we have this if statement:

if enableIngestion {
  server.rawMetricsV2 = newRawHttpHandlerV2(logger, serverName, handler)
  routes = append(routes,
		route{path: "/v2/raw", handler: server.rawMetricsV2.MetricHandler, methods: []string{"POST"}, name: "metricsv2_post"},
		route{path: "/v2/event", handler: server.rawMetricsV2.EventHandler, methods: []string{"POST"}, name: "eventsv2_post"},
	)
}

So this means if ingestion is enabled, then attach the MetricHandler and EventHandler to the routes, which then is attached to the server. I guess I can do similar things for the Prometheus Statser? But does that mean that I have to pass in additional parameters such as the Statser and a flag enablePrometheus to the NewHttpServer method to make it work?

If it is not the case, then I am not sure what you mean by checking if it's enabled and how we can attach the handler.

[tiedotguy]

You don't need to worry about Metric/EventHandler, that would be for making a prom backend (#285), this is purely for internal metrics.

I would suggest approaching this in two steps. First, implement the stats.Statser interface, exposing prom metrics without regard for pkg/web. This is just so that you can get the prom stuff working. Ideally this would be done without using the DefaultRegistry/DefaultGatherer, so as to avoid singletons.

Once you've got the metrics exposed and working that way, you need to get the http.Handler through to pkg/web. As mentioned, I recommend just passing the stats.Statser through and doing a type check on it.