Identifying origin of Alerts sent via AlertManager

[ctwilleager-alio]

For several days now I've been struggling with getting some sort of relevant identifier into Prometheus alerts that indicate what Kubernetes cluster an alert is being sent from.

Currently, the messages in our Slack channel are largely useless because while they do indicate a problem, there's zero indication of which cluster or environment that the alert originated from. Thus, the alerts are of no use as they are not actionable.

We are using Amazon EKS with Managed Node Groups and I have explicitly tagged the managed node group with a cluster label. This means that every node in the cluster that is online, or will come online due to autoscaling, will have this label attached to it. I want to use this node label in AlertManager to tag every single alert with a cluster label and display that label in the alert content so the alerts we get are actually useful.

The other caveat here is that I need to keep the configuration to a single prometheus.jsonnet file applicable to all of our clusters. I do not want to have to deal with trying to keep a dozen different prometheus.jsonnet files to cover environments as this is basically needless configuration sprawl.

So does anyone know how I can leverage a node label on my EKS cluster as an identifier I can use in AlertManager to tag my alerts so we know where a problem is?