API object deletion unreliable if HA zone is not connected

[julianbrost]

When deleting objects using the REST API while the nodes aren't connected in an HA zone results in the objects reappearing at a later time.

To Reproduce

1. Start with an HA master setup with both running.
2. Create a few objects using the API:

   for i in {1..100}; do
     curl --fail-with-body -sSku root:icinga -X PUT "https://localhost:5665/v1/objects/hosts/test-delete-$i" \
          --json '{"attrs": {"check_command":"dummy"}}'
   done

3. Check /var/lib/icinga2/api/packages/_api/*/conf.d/hosts/test-delete-*.conf on both masters to verify the objects exist on both.
4. Stop the second master.
5. Delete the just created objects via API on the first master:

   for i in {1..100}; do
     curl --fail-with-body -sSku root:icinga -X DELETE "https://localhost:5665/v1/objects/hosts/test-delete-$i" \
          --json '{"cascade": true}'
   done

6. Repeat step 3, there should be none remaining on the first master and all on the second as it's not running.
7. Start the second master again.
8. Wait a bit for the masters to sync and repeat step 3 again. This is where things start to go wrong: the objects disappeared from the second master but reappeared on the first.
9. Restart the second master.
10. Wait for them to sync and repeat step 3 again. Now the objects are also reappearing on the second master and the objects exist on both masters again.

Theory on what happens

When both masters reconnect, both generate messages to sync the other. The first master will replay the delete messages to the second and the second will send the object updates for the (from its perspective still existing) objects to the first master causing them to reappear.

Your Environment

Tested with the current master (9e31b8b).