CreatedByAnnotation is deprecated in 1.9

[chino]

See: kubernetes/kubernetes#54445

Hence the updater ends up killing it self before it gets a chance to issue the reboot. This causes it to go into an endless loop of setting the node schedulable and unschedulable and killing any pods that land on the node.

• https://github.com/coreos/container-linux-update-operator/blob/master/pkg/drain/drain.go#L38
• https://github.com/coreos/container-linux-update-operator/blob/master/pkg/agent/agent.go#L153
• https://github.com/coreos/container-linux-update-operator/blob/master/pkg/agent/agent.go#L183

To confirm I faked an annotation and it seems to be rebooting nodes now:

kubectl -n reboot-coordinator annotate pods -l app=container-linux-update-agent --overwrite kubernetes.io/created-by='{"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"DaemonSet","namespace":"reboot-coordinator","name":"container-linux-update-agent","uid":"5ef034e0-1890-11e6-8935-42010af0003e","apiVersion":"v1","resourceVersion":"427339"}}'

[dghubble]

I can reproduce this on my v1.9.1 clusters. Missed it in 1.9 testing since I manually forced updates across all clusters as part of Meltdown mitigations.

[dghubble]

Impact is serious because Kubernetes 1.9 clusters with CLUO aren't applying Container Linux updates. Open-source users and downstream Tectonic users will need to get the patched version of CLUO. Impact on workloads is mostly mitigated, pods shift as nodes are marked unscheduable one at a time.

I have an initial patch in #164

[dghubble]

Fixed in quay.io/coreos/container-linux-update-operator:v0.5.0. Thanks for the report!