kubelet does not update node labels given by --node-labels upon node upgrade

[MrHohn]

Is this a BUG REPORT or FEATURE REQUEST?:
/kind bug
/sig node

What happened:
From https://k8s-testgrid.appspot.com/sig-network-gce#gci-gce-latest-upgrade-kube-proxy-ds.

The kube-proxy daemonset migration test has been failing for a while, investigated it today and found the root cause that upon node upgrade, kubelet sometimes doesn't update node labels even if they are given with the --node-labels flag. The test failed because kube-proxy daemonset relies on kubelet applying beta.kubernetes.io/kube-proxy-ds-ready=true label upon upgrade so that it can be schedule onto the node.

Dig a bit from the kubelet logs and found a symptom that kubelet will not update node labels properly whenever this log line shows up for the node:

I0607 22:00:55.776996    1493 kubelet_node_status.go:123] Node e2e-test-XXX-YYYYY was previously registered

From a quick search on commits and found a recent change #61877, which stopped kubelet from deleting the node object and creating a new one upon upgrade. Instead, kubelet will now patch the nodeStatus for any difference. Though, surprisingly it seems like kubelet actually depends on the node object recreation logic for updating the node labels, which explains the failure symptom observed above.

@kubernetes/sig-node-bugs
cc @thockin @dcbw

What you expected to happen:

Node labels given to kubelet by --node-labels should be set on node.

How to reproduce it (as minimally and precisely as possible):
Run kube-proxy daemonset migration test (which triggers a cluster upgrade) and observe it to fail.

STORAGE_MEDIA_TYPE=application/vnd.kubernetes.protobuf go run hack/e2e.go --check-version-skew=false --test --test_args="--ginkgo.focus=KubeProxyDaemonSetMigration --ginkgo.skip=Downgrade"

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): HEAD

[liggitt]

Duplicate of #18394

See also #18307 for data loss problems caused by kubelets acting as though they own their own labels

See also kubernetes/community#911 for security/isolation problems caused by kubelets being able to self-label

/close