Why does alertmanagerconfigs automatically add a namespace matcher

[snuggie12]

What did you do?
I created an AlertmanagerConfig and noticed it is adding a matcher for namespace. Is there some way to turn this off? What if there is no namespace label? What if the namespace is under a different label? What if I want it to be for all namespaces?

Did you expect to see some different?
I think I sort of understand why this exists, but curious if there's any way to turn this off.

Environment

• Prometheus Operator version:

  v0.44.0

• Kubernetes version information:

  v1.16.15-gke.4300

• Kubernetes cluster kind:

  gke using terraform

• Manifests:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: blackhole
  namespace: monitoring
spec:
  receivers:
  - name: blackhole
  route:
    matchers:
    - name: severity
      value: info
    receiver: blackhole

• Prometheus Operator Logs:

Anything else we need to know?:

# Config that is generated by CR:
  - receiver: monitoring-blackhole-blackhole
    match:
      namespace: monitoring
      severity: info
    continue: true

[brancz]

That's kind of the point of the feature, otherwise it's possible that alertmanager configs in different namespaces conflict and Alertmanager won't be able to start.

[snuggie12]

Perhaps I'm in the minority but the feature I really cared about was using secret refs.

Regardless, do you see any reason why current behavior can't be the default and provide the ability to override?

One other thought I haven't tried is deploy only receivers via the object and, when necessary, deploy the route via the normal secret?

[kragniz]

#3737 has the same problem: being able to specify a default on the alertmanager resource would be great

[snuggie12]

Doing just receivers in the AlertmanagerConfig works as expected.

As there are no secrets in the routes (at least for me, very much a prometheus newb,) I put them back in the secret. Since we use kustomize I should be able to put the routes in plaintext in git and use a secret generator (without the hash) to create the Secret.

I suppose my issues are resolved but I have two semi-contradictory thoughts on this request:

My interpretation of the choice to make namespace a default is that a Namespace should be treated as the owner of a workload (in relation to routes.) That's how it works for most of our workloads but some of our infra stuff is mixed between teams but exist in the same namespace. Definitely making me re-think some of those design choices.

Edge cases are inevitable given so many companies. I think the ability to turn off the namespace label (while leaving it as the default) is still preferable. If I understand the proposed solution in #3737 (which I definitely might not,) putting AlertManagerConfig in the Alertmanager CRD presents 3 problems:

• I think you can only have 1 Alertmanager per alertmanager instance? Or can you create 2 and they'll concatenate together? If so, how does that work with namespaces?
• The reason I mention all that is that the AlertmanagerConfig allows for self service but if you're restricted to only 1 Alertmanager or can't combine them if they're in different namespaces we take away self service.
• Don't know if I'm in the minority but I use the helm chart to produce the Alertmanager which means more text in my values file. I would prefer to have the actual CRs checked into git instead of putting the content of the CR in helm and have a controller create the CR for me.

[snuggie12]

Actually, maybe doing only receivers only semi works? If you make a route in the Secret and you use the name for the receiver that is auto generated it returns this error:

level=error ts=2020-12-11T21:50:58.051382243Z caller=klog.go:96 component=k8s_client_runtime func=ErrorDepth msg="Sync \"monitoring/prometheus-stack-alertmanager\" failed: provision alertmanager configuration: base config from Secret could not be parsed: undefined receiver \"monitoring-cpe-receivers-cpe-critical\" used in route"

[jutley]

I also tried using the AlertmanagerConfig CRD solely for the benefit of using SecretRefs in receivers. Being able to due that would significantly simplify our Alertmanager configuration.

Currently, we generate a configmap with the config with the secrets templated, use an encrypted Secret with just the secret values, and run a small controller to bind the template with the real value, and update a Secret with the contents. All this was necessary in order to use a GitOps-based deployment process.

This new CRD doesn't work for us. Our Alertmanager configuration is centrally managed, our alerts often do not include a namespace, and we have many namespaces which should all have the same Alertmanager routes and receivers.

[snuggie12]

I've worked my way around most of it but it's proven to be tough.

• receivers only doesn't work at all (mistaken from my previous comment) It appears as if the secret is processed separately from the alertmanagerconfig resources (also possible they are individually processed.) I believe part of that process is to validate and routes reference receivers that don't exist even if you point at the altered name with the namespace in it.
• Had to blackhole Watchdog and invent my own copy of Watchdog that comes with a namespace of my choosing.
• Last item remaining is that my team is the default route and also cover vastly more namespaces than any other team since it's mostly off the shelf software. For now that secret is still a sealed secret but hopefully once this is fixed I can remove that from the secret and with no secrets present kustomize can generate it from a plaintext file for a perfect gitops solution.

[simonpasquier]

I think this is the same feature as requested in #3737?

[snuggie12]

I'd say they are the same, though I throw out some reasons against the embedded option and why a ClusterAlertmanagerConfig would be better (that being helm bloat and we have multiple ops teams that have access to multiple namespaces.)

I see some others have also spoken about the embedded stuff so I'm good to close this.

I've worked around most stuff but is there any documentation around performance and routes/receivers? Like things you should avoid (too many levels of nesting routes or something?)

[simonpasquier]

any documentation around performance and routes/receivers? Like things you should avoid (too many levels of nesting routes or something?)

No concrete numbers but in general, Alertmanager can cope with quite complex routing configuration:
https://twitter.com/timsimlol/status/1145790451129167872

[bcbrockway]

Another GitOps follower here who just wanted to use this for the secret ref functionality! Spent half a day trying to implement in jsonnet only to find I'd have to duplicate all our routes and receivers in all namespaces just so I can inject an OpsGenie API key into the config. I think we're gonna have to go back to storing the whole thing in our nasty jsonnet -> git-crypted Secret -> SealedSecret workflow now :(

Would really love the ability to do this sometime in the future.

[muffl0n]

What I found extremely frustrating and irritating: When you add an AlertmanagerConfig, it set's the namespace as a filter. But when you add a PrometheusRule to the same namespace, it does not add a corresponding label! I understand the logic behind that, but it just cost me around two days to figure it out.
Also I have to set the namespace as a label to every alert rule under spec in PrometheusRule. Or is there a way to set it per PrometheusRule?

[simonpasquier]

yes this is enforcedNamespaceLabel in Prometheus and ThanosRuler specs.

[MKruger777]

Same problem here :- ( Setting my hopes on PR 4034

[amperen]

Fully agreed with @muffl0n.

@simonpasquier enforcedNamespaceLabel helps only if all rules are created in one namespace. Accordingly, when you need to process rules from multiple namespaces, you still have to specify a specific namespace in the labels of a specific rule.

[fourstepper]

Based on what is the match directive set? Is it based on which namespace the Alertmanager configured exists in, or something else?

[xinnj]

No idea why the design like this. We have prometheusrules created from different namespaces, and we want to have a unique alertmanagerconfigs to set alert notify routes. But now , it seems it's impossible. Really ridiculous.

[snuggie12]

@xinnj I'd follow #3737. I would close this but don't know if maintainers find this useful for SEO or something.

[Shaybs]

Yes @xinnj you made a great point. What is this?

[Vladikamira]

guys, you can have a look here, VictoriaMetrics operator recently add an option to disable namespace label injection into matchers section, jfyi

[BouchaaraAdil]

this part here crd-alertmanagerconfigs.yaml#L4382-L4384 doesn't make sense:

The Alertmanager route definition for alerts matching
                  the resource’s namespace. If present, it will be added to the generated
                  Alertmanager configuration as a first-level route.

because there's some alerts without namespace labels, but with other labels and forcing resource's namespace match will lead to ignore other match conditions based on some other labels:

example, by applying below alertmanagerconfig spec:

route:
  matchers:
    - name: squad
      value: test
  receiver: test

  routes:
    - matchers:
      - name: squad
        value: test
      receiver: test
      continue: true
receivers:
  - name: test
  ...

is resulting to the below alertmanagerconfig:

route:
  receiver: slack-alerts
  group_by:
  - alertname
  - cluster
  - service
  continue: false
  routes:
  - receiver: monitoring/prometheus-operator-minikube-test/test
    match:
      squad: test
    matchers:
    - namespace="monitoring"
    continue: true
    routes:
    - receiver: monitoring/prometheus-operator-minikube-test/test
      match:
        squad: test
      continue: true
inhibit_rules:
...
receivers:
- name: monitoring/prometheus-operator-minikube-test/test
..

i want to remove the namespace matcher and keep only defined ones because there are some alerts that don't contain a namespace label.

[osttra-j-sernbrant]

Setting alertmanagerSpec.alertmanagerConfigMatcherStrategy.type: None disables the namespace matcher behaviour.

[vponoikoait]

Didn't help in my case, but #4220 seems to be related

[vponoikoait]

Created issue for this non-working in prometheus-community/helm-charts
prometheus-community/helm-charts#4343

[lud0v1c]

Thanks @osttra-j-sernbrant, works like a charm

[meysam81]

One other thing I had to do beside this matcher strategy was to leave the default Alertmanager config as is, and create my own AlertmanagerConfig with custom routes and receivers. Hope it helps someone on the internet. 🙏

[brunomgv]

Thanks @osttra-j-sernbrant , I spent a few days trying to solve that. This solution worked for me.