If RollingUpgrade object is deleted during rollout, upgrade manager doesn't stop the rollout.

[uthark]

1. It continues to rollout.
2. It fails to update object (because it is gone)

{"level":"error","ts":1605329457.7081757,"logger":"controllers.RollingUpgrade","msg":"failed to update status","rollingupgrade":"rollingupgrade-node-us-west-2a","error":"Operation cannot be fulfilled on rollingupgrades.upgrademgr.keikoproj.io \"rollingupgrade-node-us-west-2a\": StorageError: invalid object, Code: 4, Key: /registry/upgrademgr.keikoproj.io/rollingupgrades/keiko-upgrade-manager/rollingupgrade-node-us-west-2a, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 5b4944eb-f154-412d-b6cb-11c65b6580c2, UID in object meta: ","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128\ngithub.com/keikoproj/upgrade-manager/controllers.(*RollingUpgradeReconciler).error\n\t/workspace/controllers/rollingupgrade_controller.go:1132\ngithub.com/keikoproj/upgrade-manager/controllers.(*RollingUpgradeReconciler).UpdateInstance\n\t/workspace/controllers/rollingupgrade_controller.go:1061"}

Proper reconciliation should stop existing rollout and not fail.

[shrinandj]

There was some code added to handle this. However, there will always be a "time-of-check-to-time-of-use" window.

See

upgrade-manager/controllers/rollingupgrade_controller.go

Line 544 in 40b0873

// Check if the err is "StorageError: invalid object". If so, the object was deleted...

and

upgrade-manager/controllers/rollingupgrade_controller.go

Line 1027 in 40b0873

// Check if the err is "StorageError: invalid object". If so, the object was deleted...

There were several tests done when the functionality was added: #97.

Maybe, some scenario was missed?