Deleting an AlertPolicy doesn't remove the AlertsApmCondition

[AdrienKuhn]

Description

When creating an AlertPolicy containing an AlertsApmCondition, removing the policy will not remove the AlertsApmCondition from Kubernetes resources and the operator will be in an infinite error loop.

Current behavior

The AlertsApmCondition is not removed from Kubernetes cluster after removing an AlertPolicy, even if the condition doesn't exist on NewRelic anymore.

Expected behavior

The AlertsApmCondition should be removed from Kubernetes cluster after removing an AlertPolicy.

Steps To Reproduce

Steps to reproduce the behavior:

1. Deploy a policy

$ cat <<EOF | kubectl apply -f -
apiVersion: nr.k8s.newrelic.com/v1
kind: AlertsPolicy
metadata:
  name: my-alert-policy
spec:
  account_id: [account_id]
  api_key_secret:
    name: nr-api-key
    namespace: newrelic-kubernetes-operator-system
    key_name: api-key
  name: "My test policy"
  incidentPreference: "PER_CONDITION_AND_TARGET"
  region: "US"
  conditions:
    - spec:
        type: "apm_app_metric"
        enabled: true
        metric: "apdex"
        condition_scope: application
        entities:
          - "[entity_id]"
        apm_terms:
          - threshold: "0.7"
            time_function: "all"
            duration: "10"
            priority: "warning"
            operator: "below"
          - threshold: "0.7"
            time_function: "all"
            duration: "15"
            priority: "critical"
            operator: "below"
EOF

$ kubectl get AlertsPolicy
NAME              AGE
my-alert-policy   38s

kubectl get AlertsApmCondition
NAME                              CREATED
my-alert-policy-condition-2pw8d

$ kubectl delete AlertsPolicy my-alert-policy

4. In the manager logs

{"level":"error","ts":1598903639.5313737,"logger":"controllers.AlertsAPMCondition","msg":"Failed to delete API Condition","conditionId":15881394,"region":"US","Api Key":"...","error":"resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/controllers.(*AlertsAPMConditionReconciler).Reconcile\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/controllers/alerts_apmcondition_controller.go:110\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}
{"level":"info","ts":1598903639.5314548,"logger":"controllers.AlertsAPMCondition","msg":"New Relic API returned resource not found, deleting condition resource"}
{"level":"info","ts":1598903639.5314605,"logger":"controllers.AlertsAPMCondition","msg":"New Relic Alert condition deleted, Removing finalizer"}
{"level":"info","ts":1598903639.5360534,"logger":"alertsapmcondition-resource","msg":"default","name":"my-alert-policy-condition-2pw8d"}
{"level":"info","ts":1598903639.5399203,"logger":"alertsapmcondition-resource","msg":"validate update","name":{"apiVersion":"nr.k8s.newrelic.com/v1","kind":"AlertsAPMCondition","namespace":"newrelic-kubernetes-operator-system","name":"my-alert-policy-condition-2pw8d"}}
{"level":"info","ts":1598903639.5399554,"logger":"alertsapmcondition-resource","msg":"Checking existing","policyId":"1025375"}
{"level":"info","ts":1598903639.8808038,"logger":"alertsapmcondition-resource","msg":"Deleting resource","errAlertPolicy":"resource not found"}
{"level":"error","ts":1598903639.8809204,"logger":"alertsapmcondition-resource","msg":"failed to get policy","policyId":"1025375","API Key":"...","region":"US","error":"resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).CheckExistingPolicyID\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:276\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).ValidateUpdate\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:117\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/validator.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/webhook.go:135\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/http.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook.instrumentedHook.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/server.go:129\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2041\nnet/http.(*ServeMux).ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2416\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2836\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:1924"}
{"level":"info","ts":1598903639.8881001,"logger":"alertsapmcondition-resource","msg":"default","name":"my-alert-policy-condition-2pw8d"}
{"level":"info","ts":1598903639.8928833,"logger":"alertsapmcondition-resource","msg":"validate update","name":{"apiVersion":"nr.k8s.newrelic.com/v1","kind":"AlertsAPMCondition","namespace":"newrelic-kubernetes-operator-system","name":"my-alert-policy-condition-2pw8d"}}
{"level":"info","ts":1598903639.89293,"logger":"alertsapmcondition-resource","msg":"Checking existing","policyId":"1025375"}
{"level":"info","ts":1598903640.4122937,"logger":"alertsapmcondition-resource","msg":"Deleting resource","errAlertPolicy":"resource not found"}
{"level":"error","ts":1598903640.412395,"logger":"alertsapmcondition-resource","msg":"failed to get policy","policyId":"1025375","API Key":"...","region":"US","error":"resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).CheckExistingPolicyID\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:276\ngithub.com/newrelic/newrelic-kubernetes-operator/api/v1.(*AlertsAPMCondition).ValidateUpdate\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/api/v1/alerts_apmcondition_webhook.go:117\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/validator.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/webhook.go:135\nsigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/admission/http.go:87\nsigs.k8s.io/controller-runtime/pkg/webhook.instrumentedHook.func1\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/webhook/server.go:129\nnet/http.HandlerFunc.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2041\nnet/http.(*ServeMux).ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2416\nnet/http.serverHandler.ServeHTTP\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:2836\nnet/http.(*conn).serve\n\t/opt/hostedtoolcache/go/1.14.6/x64/src/net/http/server.go:1924"}
{"level":"error","ts":1598903640.414743,"logger":"controllers.AlertsAPMCondition","msg":"Failed to update condition after deleting New Relic Alert condition","error":"admission webhook \"valertsapmcondition.kb.io\" denied the request: resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/newrelic/newrelic-kubernetes-operator/controllers.(*AlertsAPMConditionReconciler).Reconcile\n\t/home/runner/work/newrelic-kubernetes-operator/newrelic-kubernetes-operator/controllers/alerts_apmcondition_controller.go:130\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:256\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}
{"level":"error","ts":1598903640.4149141,"logger":"controller-runtime.controller","msg":"Reconciler error","controller":"alertsapmcondition","request":"newrelic-kubernetes-operator-system/my-alert-policy-condition-2pw8d","error":"admission webhook \"valertsapmcondition.kb.io\" denied the request: resource not found","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/home/runner/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:258\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:232\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.0/pkg/internal/controller/controller.go:211\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/home/runner/go/pkg/mod/k8s.io/apimachinery@v0.18.4/pkg/util/wait/wait.go:90"}

5. AlertsApmCondition is still there and there is no way to delete it

$ kubectl get AlertsApmCondition
NAME                                                   CREATED
my-alert-policy-condition-2pw8d

$ kubectl edit AlertsApmCondition my-alert-policy-condition-2pw8d # Remove the `alertsapmconditions.finalizers.nr.k8s.newrelic.com` finalizer
error: alertsapmconditions.nr.k8s.newrelic.com "my-alert-policy-condition-2pw8d" could not be patched: admission webhook "valertsapmcondition.kb.io" denied the request: resource not found

[guohaolee]

Just to add a note if someone faced this issue.

You'll need to edit the existing policy id in the manifest with any policy ID that exist in your newrelic. Once done that it will be deleted.

If still not deleted, patch to remove the finalizers