Stale token in cloud-init-settings/kube-system-hetzner-kubelet-bootstrap-config

[alexMV1989]

Just installed OSM creates a secret cloud-init-settings/kube-system-hetzner-kubelet-bootstrap-config, all working fine, but after an hour cloud machines cant add to cluster due to token in /etc/kubernetes/bootstrap-kubelet.conf is wrong

I found that bootstrap token (that creates by OSM) with TTL=1hour

root@us-west2-master4:~# kubeadm token list
TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS
j5s48n.sthmG832vjrfdxlr 59m 2023-11-21T15:11:00Z authentication,signing bootstrap token for kube-system-hetzner system:bootstrappers:machine-controller:default-node-token

This token OSM pod writes to cloud-init-settings/kube-system-hetzner-kubelet-bootstrap-config
First hour everything goes fine, cloud machines creartes and working fine

After this first hour cloud machines stopped add to cluster due to kubeadm token is stale and OSM pod couldnt write new token to cloud-init-settings/kube-system-hetzner-kubelet-bootstrap-config

I see such error in OSM pod log:

{"level":"error","ts":1700571426.160011,"caller":"osc/osc_controller.go:167","msg":"Reconciling failed","error":"failed to reconcile operating system config: failed to create bootstrap kubeconfig: secrets "kube-system-hetzner-kubelet-bootstrap-config" is forbidden: User "system:serviceaccount:kube-system:operating-system-manager" cannot update resource "secrets" in API group "" in the namespace "cloud-init-settings"","stacktrace":"k8c.io/operating-system-manager/pkg/controllers/osc.(*Reconciler).Reconcile\n\tk8c.io/operating-system-manager/pkg/controllers/osc/osc_controller.go:167\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}

I found that there are no permissions to update this token: https://github.com/kubermatic/machine-controller/blob/main/examples/operating-system-manager.yaml#L1154-L1158
After I added an update verb cloud-init-settings/operating-system-manager role the secret cloud-init-settings/kube-system-hetzner-kubelet-bootstrap-config going to be updated and fixed this issue

Is it a bug? I think you need to add an update verb to role "cloud-init-settings/operating-system-manager"

[embik]

That looks like a bug indeed in the example manifests indeed. Would you be interested in submitting the change, as you correctly identified what needs to be changed?

[kubermatic-bot]

Issues go stale after 90d of inactivity.
After a furter 30 days, they will turn rotten.
Mark the issue as fresh with /remove-lifecycle stale.

If this issue is safe to close now please do so with /close.

/lifecycle stale

[kubermatic-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

/lifecycle rotten

[embik]

/remove-lifecycle rotten