fix: label mismatch for AlertmanagerFailedToSendAlerts

[cr7258]

Description

Since Alertmanager 0.26.0 version, a new label reason was added in the alertmanager_notifications_failed_total metric to indicate the type of error of the alert delivery.

As a result, the original alert rules are broken because labels are mismatched between alertmanager_notifications_failed_total and alertmanager_notifications_total metrics.
Prometheus requires samples with exactly the same labels to get matched together when performing calculations. docs

Use the ignoring vector matching keyword to ignore the new reason label to allow for matching between series with different labels.

Type of change

What type of changes does your code introduce to the kube-prometheus? Put an x in the box that apply.

• CHANGE (fix or feature that would cause existing functionality to not work as expected)
• FEATURE (non-breaking change which adds functionality)
• BUGFIX (non-breaking change which fixes an issue)
• ENHANCEMENT (non-breaking change which improves existing functionality)
• NONE (if none of the other choices apply. Example, tooling, build system, CI, docs, etc.)

Changelog entry

Please put a one-line changelog entry below. Later this will be copied to the changelog file.

Fix label mismatch for AlertmanagerFailedToSendAlerts

[simonpasquier]

Thanks for spotting it! The fix needs to happen in prometheus/alertmanager first though.

[cr7258]

@simonpasquier The PR has been merged in prometheus/alertmanager repo. Can we proceed with this PR now?

[philipgough]

I think we will need a release in AM first.