Automatic update process (production interruption)

[chr4]

Describe the bug

We've experienced some issues with the automatic (security) updates of keepalived. This happens when installing the keepalived package either via Ubuntu packages (having automatic security updates activated) or when installing the snap package (as recommended iirc), which has automatic updates that apparently cant' be disabled. When the update is triggered, it results in a restart of the keepalived daemon (not reload), which causes a failover (we currently do not use nopreempt).

• What's the recommended way of handling automatic (security) updates?
• Is there a posibility to configure keepalived in a way that doesn't failover upon restarts?

To Reproduce

Install keepalived and wait for an update

Expected behavior

The update process should never hard-restart the daemon, unless it can be done so without any interuption of services.

Keepalived version

• Keepalived v1.3.9 (10/21,2017) (Ubuntu apt package)
• v2.0.16-11-g1b96509+ (from snap)

Distro (please complete the following information):

Ubuntu 18.04 LTS (bionic beaver)

Did keepalived coredump?

No.

[pqarmitage]

From what you have written, it appears that after the security update has been installed, the system wants to run the new package rather than the old one (which is quite reasonable for security updates). This means that the old process has to terminate and the new process start running (hence the restart rather than the reload). nopreempt will make no difference to this.

A restart means that keepalived stops, and is then started again. When keepalived stops, any instance that is master will send priority 0 adverts, which start a fast take-over by the highest priority backup (less than 1 advert interval). When keepalived starts up again, it waits three advert intervals (plus a bit more) to see if there is a higher priority master, and only if it sees no higher priority adverts will it take over as master (this is all in accordance with the VRRP RFCs).

You ask two questions:

• What's the recommended way of handling automatic (security) updates?

I don't think keepalived has any recommendations on this. Security updates are a distro feature, and so I think if there are any recommendations, they would have to come from the distro provider.

• Is there a possibility to configure keepalived in a way that doesn't failover upon restarts?

At the moment, the answer is no.

The only way I can see of doing this is if keepalived is told to reload its processes. Keepalived would need to save the state of each keepalived instance and when the new process starts, restore those states immediately. This is doable but would take a bit of work.

[chr4]

I don't think keepalived has any recommendations on this. Security updates are a distro feature, and so I think if there are any recommendations, they would have to come from the distro provider.

I understand. I'm asking this, because this comment seemed to recommend using snap to install keepalived on Ubuntu. With the information you just gave plus the fact, that autoupdates in snap can't be disabled, this will inevitably result in downtimes which can't be prevented. This would makes snap unfit for use in high-available environments. So, the only stable way of running recent versions of keepalived would be to compile it from source? Or am I missing another option here?

The only way I can see of doing this is if keepalived is told to reload its processes. Keepalived would need to save the state of each keepalived instance and when the new process starts, restore those states immediately. This is doable but would take a bit of work.

This sounds good, maybe that'd be a way to go to fix this in the long run. Apprechiate your work!