Alertmanager multi tenancy (Rancher 2.79)

[Freshwood]

Hello community,

we have spent multiple days to figure out how to have a working monitoring / alerting system.

Actually we hit the issue #3737
It is impossible to send alerts from an different namespace as the monitoring namespace.

So we decided to setup a dedicated alert manager inside a project namespace. Example:

apiVersion: monitoring.coreos.com/v1
kind: Alertmanager
metadata:
  name: alert-manager
spec:
  alertmanagerConfigSelector:
    matchLabels:
      alertmanagerConfig: alert-manager-config
  replicas: 1

With alert manager config:

apiVersion: monitoring.coreos.com/v1alpha1
kind: AlertmanagerConfig
metadata:
  name: alert-manager-config
  labels:
    alertmanagerConfig: alert-manager-config
spec:
  route:
#    groupBy: [ job ]
    matchers:
      - name: severity
        value: warning
#    groupWait: 10s
#    groupInterval: 5m
#    repeatInterval: 3h
    receiver: 'test-alert'

  receivers:
    - name: 'test-alert'
      webhookConfigs:
        - url: http://test-service.namespace.svc.cluster.local:5678
          sendResolved: true

With a prometheus rule in the project namespace which will fire:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: prometheus-rule
spec:
  groups:
    - name: example.rules
      rules:
        - alert: ExampleAlert
          expr: prometheus_build_info{version="2.38.0"} == 1
#          for: 5s
          labels:
            severity: warning
          annotations:
            summary: "This is a test alert"
            description: "The is just a test alert"

I was inspired by the alerting guide https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/alerting.md but could not figure out how to setup a multi tenant alertmanager installation.

Does anybody know if this is possible or are there any further documentation about this topic?

Regards,

Tobias