Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloud Controller Manager doesn't add droplets to Load Balancer #639

Closed
lev-stas opened this issue Jul 4, 2023 · 9 comments
Closed

Cloud Controller Manager doesn't add droplets to Load Balancer #639

lev-stas opened this issue Jul 4, 2023 · 9 comments

Comments

@lev-stas
Copy link

lev-stas commented Jul 4, 2023

Hi! I have deployed k8s cluster on DO droplets and I am using digitalocean-cloud-controller-manager in it. When I am creating nginx ingress controller, DO load balancer created automatically, but droplets, where k8s hosted are not added to it automatically. I see warning message in digitalocean-cloud-controller-manager pod with such content

type="Warning" reason="UnAvailableLoadBalancer" message="There are no available nodes for LoadBalancer"

I have deployed cloud controller manager with all recomendations. I set --cloud-provider=external in kubelet, and --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname in kube-api server. I also tried to add and remove --provider-id option to kubelet, but it didn't have any effect.

When I add droplets to Load Balancer manually, everything is working OK, and I cat manage traffic to necessary services. But in case of any changes in LoadBalacer type service in k8s all settings are dropped and I should do manual adding again.

I saw several issues related to this problems, but they don't have any certain recipes to solve it.
@timoreimann
Copy link
Collaborator

The message indicates that nodes are either not there at all or in a state that does not allow them to be added, namely being unready. Can you check on that? And if it's looking good, could you share the YAML output of a node that you think should be added but isn't?

@lev-stas
Copy link
Author

lev-stas commented Jul 5, 2023 •
edited

I have a simple cluster with one controlplane and one worker node. All nodes are in ready status

stas@ivinco-k8s-controlplane01:~$ kubectl get nodes
NAME                        STATUS   ROLES           AGE   VERSION
ivinco-k8s-controlplane01   Ready    control-plane   40h   v1.27.3
ivinco-k8s-worker01         Ready    <none>          40h   v1.27.3
stas@ivinco-k8s-controlplane01:~$

and I can run pods normally. When I've added droplets to loadbalancer manually, the application is available from outside world. Here is the the yaml file of worker node, which should be definitely attached to loadbalancer automatically

apiVersion: v1
kind: Node
metadata:
  annotations:
    alpha.kubernetes.io/provided-node-ip: 10.114.0.2
    csi.volume.kubernetes.io/nodeid: '{"csi.tigera.io":"ivinco-k8s-worker01","driver.longhorn.io":"ivinco-k8s-worker01"}'
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
    node.alpha.kubernetes.io/ttl: "0"
    projectcalico.org/IPv4Address: 10.114.0.2/20
    projectcalico.org/IPv4VXLANTunnelAddr: 192.168.205.0
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2023-07-03T18:34:32Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: ivinco-k8s-worker01
    kubernetes.io/os: linux
  name: ivinco-k8s-worker01
  resourceVersion: "406517"
  uid: 3de39ee4-3584-4cda-abfa-2e86e29c14bf
spec:
  podCIDR: 192.168.1.0/24
  podCIDRs:
  - 192.168.1.0/24
status:
  addresses:
  - address: 10.114.0.2
    type: InternalIP
  - address: ivinco-k8s-worker01
    type: Hostname
  - address: 161.35.218.29
    type: ExternalIP
  allocatable:
    cpu: "2"
    ephemeral-storage: "56911881737"
    hugepages-2Mi: "0"
    memory: 1912080Ki
    pods: "110"
  capacity:
    cpu: "2"
    ephemeral-storage: 60306Mi
    hugepages-2Mi: "0"
    memory: 2014480Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2023-07-04T19:09:13Z"
    lastTransitionTime: "2023-07-04T19:09:13Z"
    message: Calico is running on this node
    reason: CalicoIsUp
    status: "False"
    type: NetworkUnavailable
  - lastHeartbeatTime: "2023-07-05T10:34:37Z"
    lastTransitionTime: "2023-07-04T19:09:08Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2023-07-05T10:34:37Z"
    lastTransitionTime: "2023-07-04T19:09:08Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2023-07-05T10:34:37Z"
    lastTransitionTime: "2023-07-04T19:09:08Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2023-07-05T10:34:37Z"
    lastTransitionTime: "2023-07-04T19:09:08Z"
    message: kubelet is posting ready status. AppArmor enabled
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - docker.io/longhornio/longhorn-engine@sha256:32170c96be28e47c9a66c85cca65c5df8637f3d2b0a1b2034c02ec186ac6996a
    - docker.io/longhornio/longhorn-engine:v1.4.2
    sizeBytes: 267482077
  - names:
    - docker.io/longhornio/longhorn-instance-manager@sha256:9157fe047fba170c70ac5ce83d5cc9e2fcdcd3b7a73885efb2ce786c2c69bd1d
    - docker.io/longhornio/longhorn-instance-manager:v1.4.2
    sizeBytes: 266275737
  - names:
    - registry.k8s.io/ingress-nginx/controller@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd
    sizeBytes: 113903192
  - names:
    - registry.k8s.io/ingress-nginx/controller@sha256:744ae2afd433a395eeb13dc03d3313facba92e96ad71d9feaafc85925493fee3
    sizeBytes: 111174024
  - names:
    - docker.io/longhornio/longhorn-manager@sha256:b318015eee458b08c2fc8a38fd5bd947cb9df2dd747c20b5ddb38163f131a732
    - docker.io/longhornio/longhorn-manager:v1.4.2
    sizeBytes: 98278778
  - names:
    - docker.io/calico/cni@sha256:3be3c67ddba17004c292eafec98cc49368ac273b40b27c8a6621be4471d348d6
    - docker.io/calico/cni:v3.26.1
    sizeBytes: 93375656
  - names:
    - docker.io/calico/node@sha256:8e34517775f319917a0be516ed3a373dbfca650d1ee8e72158087c24356f47fb
    - docker.io/calico/node:v3.26.1
    sizeBytes: 86592218
  - names:
    - docker.io/longhornio/longhorn-ui@sha256:90b33eae3a3c5a1c932ae49589d0b0d9a33e88f55f00241e8c0cb175d819e031
    - docker.io/longhornio/longhorn-ui:v1.4.2
    sizeBytes: 72622024
  - names:
    - docker.io/library/httpd@sha256:7d45cb2af5484a1be9fdfc27ab5ada4fb8d23efdd0e34e7dd0aa994438c9f07b
    - docker.io/library/httpd:latest
    sizeBytes: 64694091
  - names:
    - docker.io/library/httpd@sha256:af30bcefc95cd5189701651fd9c710d411dc4b8536f93ef542cfa6f1fa32cb53
    sizeBytes: 64693977
  - names:
    - docker.io/paulbouwer/hello-kubernetes@sha256:2ad94733189b30844049caed7e17711bf11ed9d1116eaf10000586081451690b
    - docker.io/paulbouwer/hello-kubernetes:1.10
    sizeBytes: 45466760
  - names:
    - docker.io/digitalocean/digitalocean-cloud-controller-manager@sha256:28b7564201b0c04620c7c9b9daf901a3a9ce688f072fd69d3a56d15f157f76fd
    - docker.io/digitalocean/digitalocean-cloud-controller-manager:v0.1.43
    sizeBytes: 35515603
  - names:
    - registry.k8s.io/kube-proxy@sha256:fb2bd59aae959e9649cb34101b66bb3c65f61eee9f3f81e40ed1e2325c92e699
    - registry.k8s.io/kube-proxy:v1.27.3
    sizeBytes: 23897400
  - names:
    - docker.io/longhornio/csi-snapshotter@sha256:a3c71d6b1ecaefe69330312326b2ccf016a83af67e9b2bc2ad0685dc388c35df
    - docker.io/longhornio/csi-snapshotter:v5.0.1
    sizeBytes: 22163656
  - names:
    - docker.io/longhornio/csi-attacher@sha256:365b6e745404cf3eb39ee9bcc55b1af3c4f5357e44dddd0270c3cbb06b252a64
    - docker.io/longhornio/csi-attacher:v3.4.0
    sizeBytes: 22084988
  - names:
    - docker.io/longhornio/csi-resizer@sha256:5ce3edf2ab0167a76d09d949a644677ec42428e2b735f6ebabab76a2b3432475
    - docker.io/longhornio/csi-resizer:v1.3.0
    sizeBytes: 21671030
  - names:
    - docker.io/longhornio/csi-provisioner@sha256:daa7828fd45897fdeec17d6e987bc54cdc0a0928a7ba6b62d8f8496d6aa5d3be
    - docker.io/longhornio/csi-provisioner:v2.1.2
    sizeBytes: 21220203
  - names:
    - registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b
    sizeBytes: 20266348
  - names:
    - docker.io/library/nginx@sha256:5e1ccef1e821253829e415ac1e3eafe46920aab0bf67e0fe8a104c57dbfffdf7
    - docker.io/library/nginx:stable-alpine
    sizeBytes: 16844563
  - names:
    - docker.io/calico/node-driver-registrar@sha256:fb4dc4863c20b7fe1e9dd5e52dcf004b74e1af07d783860a08ddc5c50560f753
    - docker.io/calico/node-driver-registrar:v3.26.1
    sizeBytes: 10956618
  - names:
    - docker.io/longhornio/csi-node-driver-registrar@sha256:b3288cdcb832c30acc90f0d6363fc681880f46448f0175078bbd07f2a178634e
    - docker.io/longhornio/csi-node-driver-registrar:v2.5.0
    sizeBytes: 9132327
  - names:
    - docker.io/calico/csi@sha256:4e05036f8ad1c884ab52cae0f1874839c27c407beaa8f008d7a28d113ad9e5ed
    - docker.io/calico/csi:v3.26.1
    sizeBytes: 8910887
  - names:
    - docker.io/longhornio/livenessprobe@sha256:4a8917674f9eb175ad7d9339d8a5bc0f6f1376612f9abfc3854e7e7282bde75d
    - docker.io/longhornio/livenessprobe:v2.8.0
    sizeBytes: 8892153
  - names:
    - docker.io/calico/pod2daemon-flexvol@sha256:2aefd77a4f8289c88cfe24c0db38822de3132292d1ea4ac9192abc9583e4b54c
    - docker.io/calico/pod2daemon-flexvol:v3.26.1
    sizeBytes: 7291173
  - names:
    - docker.io/library/alpine@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69
    - docker.io/library/alpine:3.12
    sizeBytes: 2811698
  - names:
    - registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d
    - registry.k8s.io/pause:3.8
    sizeBytes: 311286
  nodeInfo:
    architecture: amd64
    bootID: 2dce3c79-5153-4279-bf7f-5b5365f5ec0d
    containerRuntimeVersion: containerd://1.7.2
    kernelVersion: 6.1.0-9-amd64
    kubeProxyVersion: v1.27.3
    kubeletVersion: v1.27.3
    machineID: 1775e2c46bb860db6e8b17d764a2f784
    operatingSystem: linux
    osImage: Debian GNU/Linux 12 (bookworm)
    systemUUID: 1992b643-962e-49e3-9528-00041357bb0a

I wonder if it is needed to add any labels to k8s nodes to be managed by digitalocean cloud controller manager, but didn't find any information in official documentation related to this issue.

@timoreimann
Copy link
Collaborator

@lev-stas you shouldn't have to add any labels. There is actually a label to keep CCM from considering a node for inclusion in the LB rotation, but that's not what you seem to have set.

There can be other conditions though that determine if a node is included or not. The corresponding filtering logic starts here and then calls into this method. The first feature gate StableLoadBalancerNodeSet checked is beta since 1.27 and thus probably enabled, making this list of sub-conditions apply which I went through one at a time. Your shared node does not seem to have a deletion timestamp, is not excluded by label, and is not tainted. However, you are apparently missing the provider ID which is a field right under .spec looking like this:

providerID: digitalocean://<droplet ID>

The provider ID must either be set explicitly on the kubelet or discovered by CCM through the DO API. What's a bit odd to me is that the node shouldn't be considered done provisioning by CCM if the provider ID has not been discovered, but in that case it should still have an "uninitialized" taint which I don't see. 🤔 So I'm not quite sure how the node could have reached its current state, but maybe these discoveries help you already somehow.

@lev-stas
Copy link
Author

lev-stas commented Jul 6, 2023 •
edited

@timoreimann thank you for your answer. As I mentioned in the init comment, I've tried both variant with --provider-id option and without it with the same result. And now I see, that I have added this option to kubelet

stas@ivinco-k8s-worker01:~$ ps aux | grep kubelet
root       49579  3.0  3.9 1631188 80056 ?       Ssl  13:59   0:04 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --node-ip=10.114.0.2 --cloud-provider=external --provider-id=digitalocean://363409977 --config=/var/lib/kubelet/config.yaml --container-runtime-endpoint=unix:///var/run/containerd/containerd.sock --pod-infra-container-image=registry.k8s.io/pause:3.9

But there is no providerID field in the node manifest.

apiVersion: v1
kind: Node
metadata:
  annotations:
    alpha.kubernetes.io/provided-node-ip: 10.114.0.2
    csi.volume.kubernetes.io/nodeid: '{"csi.tigera.io":"ivinco-k8s-worker01","driver.longhorn.io":"ivinco-k8s-worker01"}'
    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
    node.alpha.kubernetes.io/ttl: "0"
    projectcalico.org/IPv4Address: 10.114.0.2/20
    projectcalico.org/IPv4VXLANTunnelAddr: 192.168.205.0
    volumes.kubernetes.io/controller-managed-attach-detach: "true"
  creationTimestamp: "2023-07-03T18:34:32Z"
  labels:
    beta.kubernetes.io/arch: amd64
    beta.kubernetes.io/os: linux
    kubernetes.io/arch: amd64
    kubernetes.io/hostname: ivinco-k8s-worker01
    kubernetes.io/os: linux
  name: ivinco-k8s-worker01
  resourceVersion: "678507"
  uid: 3de39ee4-3584-4cda-abfa-2e86e29c14bf
spec:
  podCIDR: 192.168.1.0/24
  podCIDRs:
  - 192.168.1.0/24
status:
  addresses:
  - address: 10.114.0.2
    type: InternalIP
  - address: ivinco-k8s-worker01
    type: Hostname
  - address: 161.35.218.29
    type: ExternalIP
  allocatable:
    cpu: "2"
    ephemeral-storage: "56911881737"
    hugepages-2Mi: "0"
    memory: 1912064Ki
    pods: "110"
  capacity:
    cpu: "2"
    ephemeral-storage: 60306Mi
    hugepages-2Mi: "0"
    memory: 2014464Ki
    pods: "110"
  conditions:
  - lastHeartbeatTime: "2023-07-06T13:03:37Z"
    lastTransitionTime: "2023-07-06T13:03:37Z"
    message: Calico is running on this node
    reason: CalicoIsUp
    status: "False"
    type: NetworkUnavailable
  - lastHeartbeatTime: "2023-07-06T13:03:29Z"
    lastTransitionTime: "2023-07-06T13:03:29Z"
    message: kubelet has sufficient memory available
    reason: KubeletHasSufficientMemory
    status: "False"
    type: MemoryPressure
  - lastHeartbeatTime: "2023-07-06T13:03:29Z"
    lastTransitionTime: "2023-07-06T13:03:29Z"
    message: kubelet has no disk pressure
    reason: KubeletHasNoDiskPressure
    status: "False"
    type: DiskPressure
  - lastHeartbeatTime: "2023-07-06T13:03:29Z"
    lastTransitionTime: "2023-07-06T13:03:29Z"
    message: kubelet has sufficient PID available
    reason: KubeletHasSufficientPID
    status: "False"
    type: PIDPressure
  - lastHeartbeatTime: "2023-07-06T13:03:29Z"
    lastTransitionTime: "2023-07-06T13:03:29Z"
    message: kubelet is posting ready status. AppArmor enabled
    reason: KubeletReady
    status: "True"
    type: Ready
  daemonEndpoints:
    kubeletEndpoint:
      Port: 10250
  images:
  - names:
    - docker.io/longhornio/longhorn-engine@sha256:32170c96be28e47c9a66c85cca65c5df8637f3d2b0a1b2034c02ec186ac6996a
    - docker.io/longhornio/longhorn-engine:v1.4.2
    sizeBytes: 267482077
  - names:
    - docker.io/longhornio/longhorn-instance-manager@sha256:9157fe047fba170c70ac5ce83d5cc9e2fcdcd3b7a73885efb2ce786c2c69bd1d
    - docker.io/longhornio/longhorn-instance-manager:v1.4.2
    sizeBytes: 266275737
  - names:
    - registry.k8s.io/ingress-nginx/controller@sha256:e5c4824e7375fcf2a393e1c03c293b69759af37a9ca6abdb91b13d78a93da8bd
    sizeBytes: 113903192
  - names:
    - registry.k8s.io/ingress-nginx/controller@sha256:744ae2afd433a395eeb13dc03d3313facba92e96ad71d9feaafc85925493fee3
    sizeBytes: 111174024
  - names:
    - docker.io/longhornio/longhorn-manager@sha256:b318015eee458b08c2fc8a38fd5bd947cb9df2dd747c20b5ddb38163f131a732
    - docker.io/longhornio/longhorn-manager:v1.4.2
    sizeBytes: 98278778
  - names:
    - docker.io/calico/cni@sha256:3be3c67ddba17004c292eafec98cc49368ac273b40b27c8a6621be4471d348d6
    - docker.io/calico/cni:v3.26.1
    sizeBytes: 93375656
  - names:
    - docker.io/calico/node@sha256:8e34517775f319917a0be516ed3a373dbfca650d1ee8e72158087c24356f47fb
    - docker.io/calico/node:v3.26.1
    sizeBytes: 86592218
  - names:
    - docker.io/longhornio/longhorn-ui@sha256:90b33eae3a3c5a1c932ae49589d0b0d9a33e88f55f00241e8c0cb175d819e031
    - docker.io/longhornio/longhorn-ui:v1.4.2
    sizeBytes: 72622024
  - names:
    - docker.io/library/httpd@sha256:7d45cb2af5484a1be9fdfc27ab5ada4fb8d23efdd0e34e7dd0aa994438c9f07b
    - docker.io/library/httpd:latest
    sizeBytes: 64694091
  - names:
    - docker.io/library/httpd@sha256:af30bcefc95cd5189701651fd9c710d411dc4b8536f93ef542cfa6f1fa32cb53
    sizeBytes: 64693977
  - names:
    - docker.io/paulbouwer/hello-kubernetes@sha256:2ad94733189b30844049caed7e17711bf11ed9d1116eaf10000586081451690b
    - docker.io/paulbouwer/hello-kubernetes:1.10
    sizeBytes: 45466760
  - names:
    - docker.io/digitalocean/digitalocean-cloud-controller-manager@sha256:28b7564201b0c04620c7c9b9daf901a3a9ce688f072fd69d3a56d15f157f76fd
    - docker.io/digitalocean/digitalocean-cloud-controller-manager:v0.1.43
    sizeBytes: 35515603
  - names:
    - registry.k8s.io/kube-proxy@sha256:fb2bd59aae959e9649cb34101b66bb3c65f61eee9f3f81e40ed1e2325c92e699
    - registry.k8s.io/kube-proxy:v1.27.3
    sizeBytes: 23897400
  - names:
    - docker.io/longhornio/csi-snapshotter@sha256:a3c71d6b1ecaefe69330312326b2ccf016a83af67e9b2bc2ad0685dc388c35df
    - docker.io/longhornio/csi-snapshotter:v5.0.1
    sizeBytes: 22163656
  - names:
    - docker.io/longhornio/csi-attacher@sha256:365b6e745404cf3eb39ee9bcc55b1af3c4f5357e44dddd0270c3cbb06b252a64
    - docker.io/longhornio/csi-attacher:v3.4.0
    sizeBytes: 22084988
  - names:
    - docker.io/longhornio/csi-resizer@sha256:5ce3edf2ab0167a76d09d949a644677ec42428e2b735f6ebabab76a2b3432475
    - docker.io/longhornio/csi-resizer:v1.3.0
    sizeBytes: 21671030
  - names:
    - docker.io/longhornio/csi-provisioner@sha256:daa7828fd45897fdeec17d6e987bc54cdc0a0928a7ba6b62d8f8496d6aa5d3be
    - docker.io/longhornio/csi-provisioner:v2.1.2
    sizeBytes: 21220203
  - names:
    - registry.k8s.io/ingress-nginx/kube-webhook-certgen@sha256:543c40fd093964bc9ab509d3e791f9989963021f1e9e4c9c7b6700b02bfb227b
    sizeBytes: 20266348
  - names:
    - docker.io/library/nginx@sha256:5e1ccef1e821253829e415ac1e3eafe46920aab0bf67e0fe8a104c57dbfffdf7
    - docker.io/library/nginx:stable-alpine
    sizeBytes: 16844563
  - names:
    - docker.io/calico/node-driver-registrar@sha256:fb4dc4863c20b7fe1e9dd5e52dcf004b74e1af07d783860a08ddc5c50560f753
    - docker.io/calico/node-driver-registrar:v3.26.1
    sizeBytes: 10956618
  - names:
    - docker.io/longhornio/csi-node-driver-registrar@sha256:b3288cdcb832c30acc90f0d6363fc681880f46448f0175078bbd07f2a178634e
    - docker.io/longhornio/csi-node-driver-registrar:v2.5.0
    sizeBytes: 9132327
  - names:
    - docker.io/calico/csi@sha256:4e05036f8ad1c884ab52cae0f1874839c27c407beaa8f008d7a28d113ad9e5ed
    - docker.io/calico/csi:v3.26.1
    sizeBytes: 8910887
  - names:
    - docker.io/longhornio/livenessprobe@sha256:4a8917674f9eb175ad7d9339d8a5bc0f6f1376612f9abfc3854e7e7282bde75d
    - docker.io/longhornio/livenessprobe:v2.8.0
    sizeBytes: 8892153
  - names:
    - docker.io/calico/pod2daemon-flexvol@sha256:2aefd77a4f8289c88cfe24c0db38822de3132292d1ea4ac9192abc9583e4b54c
    - docker.io/calico/pod2daemon-flexvol:v3.26.1
    sizeBytes: 7291173
  - names:
    - docker.io/library/alpine@sha256:c75ac27b49326926b803b9ed43bf088bc220d22556de1bc5f72d742c91398f69
    - docker.io/library/alpine:3.12
    sizeBytes: 2811698
  - names:
    - registry.k8s.io/pause@sha256:9001185023633d17a2f98ff69b6ff2615b8ea02a825adffa40422f51dfdcde9d
    - registry.k8s.io/pause:3.8
    sizeBytes: 311286
  nodeInfo:
    architecture: amd64
    bootID: e74facdc-7e82-4aae-9dbb-ba9fb3693791
    containerRuntimeVersion: containerd://1.7.2
    kernelVersion: 6.1.0-10-amd64
    kubeProxyVersion: v1.27.3
    kubeletVersion: v1.27.3
    machineID: 1775e2c46bb860db6e8b17d764a2f784
    operatingSystem: linux
    osImage: Debian GNU/Linux 12 (bookworm)
    systemUUID: 1992b643-962e-49e3-9528-00041357bb0a

I have added this option to KUBELET_KUBECONFIG_ARGS env in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file. Since I see this option in the describing of kubelet service, I thought it was applied. But maybe I do it in a wrong way. There is no explicit information in the documentation about how I should add this option.

@timoreimann
Copy link
Collaborator

@lev-stas FWIW, for our managed Kubernetes offering DOKS we specify the providerID field via the KubeletConfiguration manifest/file, which is working fine for us. Maybe you want to give that a try.

But even if you don't specify it, CCM should discover it as I mentioned. If you could share the CCM logs for a newly bootstrapping node (ideally one where the provider ID is specified and one where it isn't), then we could try to look for any messages indicating why things aren't working.

@lev-stas
Copy link
Author

lev-stas commented Jul 6, 2023

@timoreimann I've added providerID option to /var/lib/kubelet/config.yaml config. And it is still not displayed in node manifest.
I have the only one worker node, so here is a log output from CMM pod.

stas@ivinco-k8s-controlplane01:~$ kubectl logs digitalocean-cloud-controller-manager-d484f8d4b-lt8tn -n kube-system
I0706 14:23:55.428601       1 serving.go:348] Generated self-signed cert in-memory
I0706 14:23:55.922382       1 serving.go:348] Generated self-signed cert in-memory
W0706 14:23:55.922491       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0706 14:23:56.293878       1 controllermanager.go:167] Version: v0.0.0-master+$Format:%H$
I0706 14:23:56.296674       1 secure_serving.go:210] Serving securely on [::]:10258
I0706 14:23:56.298595       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0706 14:23:56.420100       1 cloud.go:193] Nothing to manage since firewall name was not provided
I0706 14:23:56.420287       1 resources.go:126] No cluster ID configured -- skipping cluster dependent syncers.
I0706 14:23:56.421020       1 controllermanager.go:336] Started "cloud-node"
I0706 14:23:56.421640       1 node_controller.go:161] Sending events to api server.
I0706 14:23:56.421779       1 node_controller.go:170] Waiting for informer caches to sync
I0706 14:23:56.422039       1 controllermanager.go:336] Started "cloud-node-lifecycle"
I0706 14:23:56.422103       1 node_lifecycle_controller.go:113] Sending events to api server
I0706 14:23:56.423461       1 controllermanager.go:336] Started "service"
W0706 14:23:56.423475       1 core.go:111] --configure-cloud-routes is set, but cloud provider does not support routes. Will not configure cloud provider routes.
W0706 14:23:56.423481       1 controllermanager.go:324] Skipping "route"
I0706 14:23:56.423572       1 controller.go:229] Starting service controller
I0706 14:23:56.423590       1 shared_informer.go:311] Waiting for caches to sync for service
I0706 14:23:56.523641       1 shared_informer.go:318] Caches are synced for service
I0706 14:23:56.524040       1 event.go:307] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuringLoadBalancer" message="Ensuring load balancer"
I0706 14:23:56.524059       1 event.go:307] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Warning" reason="UnAvailableLoadBalancer" message="There are no available nodes for LoadBalancer"
I0706 14:23:58.373619       1 event.go:307] "Event occurred" object="ingress-nginx/ingress-nginx-controller" fieldPath="" kind="Service" apiVersion="v1" type="Normal" reason="EnsuredLoadBalancer" message="Ensured load balancer"
stas@ivinco-k8s-controlplane01:~$

The message about message="There are no available nodes for LoadBalancer" persists either with set providerID option and without it.

Controlplane node is tainted as NoSchedule, so I have the CMM pod only on worker node. Is it correct set up?

@timoreimann
Copy link
Collaborator

Hey @lev-stas , I think I missed to update this issue, sorry about that.

Before digging back into this one: do you still have the problem you described?

@lev-stas
Copy link
Author

I have solved the issue by adding string

...
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external --provider-id=digitalocean://<droplet ID>"
...

to the file /etc/systemd/system/kubelet.service.d/10-kubeadm.conf . So, I think the issue can be closed.

@timoreimann
Copy link
Collaborator

Thanks for confirming.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@timoreimann @lev-stas