Sharing Prometheus Rules

[coppolaop]

Would be a nice idea to share some Prometheus rules with the community.
Something like this site does with Redis, Nginx, MongoDB and Others.

[beorn7]

I assume you are referring to rules for the Go and process metrics?

For the former, we could perhaps have a "Go mixin" here similar to all the other mixins here: https://github.com/monitoring-mixins/docs/blob/master/directory.md
A mixin would bundle recording and alerting rules with Grafana dashboards.

Something similar could be done for the process metrics, but they exist across instrumentation libraries, so we might want to find an overarching location for a "process mixin".

[ArthurSens]

Maybe kube-prometheus could be a good home for "process mixin"?

wdyt @paulfantom @dgrisonnet?

[paulfantom]

For the "go mixin" we could potentially resurrect #808. After its completion, we should add it to https://monitoring.mixins.dev/ for better discovery.

As for the process mixin, I don't have a good candidate for the location. Kube-prometheus doesn't really fit 100% as the project is not aimed at generic process instrumentation. That said we could start prototyping something there and potentially move it later.

[dgrisonnet]

I agree with @paulfantom that kube-prometheus isn't a really good fit here.

I think that the general goal with mixins is to have them as close as possible to the project that is responsible for the metrics they use. Since both Go and process metrics come from instrumentation libraries, I would expect their related mixins to be hosted in the libraries repository. Do we have any guarantees that other instrumentation libraries will provide the same process metrics as the ones exposed by client_golang? If not, I think it would be better to host them in client_golang directly.

Also, having a commonplace for process mixins would lead to some issues if new process metrics are ever added. In that particular scenario will we be able to update the existing mixins without impacting client libraries that are lagging behind the new process collector update? By tying the mixin to client_golang version we will never face this kind of issue.

[ArthurSens]

Regarding #808 I think we'll need approval from the current maintainers of the project since @beorn7 has stepped down, right? (Pardon me if that was not the case)

I'm far from being a go expert, so I'd be happy to tackle it if the only thing left was converting the JSON to grafonnet 🙂.

And you both have good points regarding the "process mixin". Let's not host it on kube-prometheus

[beorn7]

Regarding #808 I think we'll need approval from the current maintainers of the project since @beorn7 has stepped down, right?

Yes, the "decision makers" here are @bwplotka and @kakkoyun.

[coppolaop]

Examples to rules stored here:

Something like this:

[paulfantom]

@coppolaop Such a site already exists at https://monitoring.mixins.dev. The prerequisite is to write a jsonnet monitoring mixin (preferably in a repo that has code for the application which exposes metrics used in alerts, ex. some exporter) and add it to this list: https://github.com/monitoring-mixins/website/blob/master/mixins.json. Next, automation picks up data and refreshes the site every day.

There is a stale PR to make [mixtool](https://github.com/monitoring-mixins/mixtool/pull/32) capable of downloading mixins from the mentioned site so installation would be easier.

FYI, the site you are referring to (https://awesome-prometheus-alerts.grep.to/) is also taking alerts from said mixins. It just doesn't offer the level of customization mixins do.