Add tolerations to existing osd's

[a1994sc]

I have been using rook for a small, 3-4 node, cluster and I want to expand it by adding some nodes... However I would like to split the load between infrastructure, rook, kyverno, etc, on the original nodes, and the other applications, harbor, keycloak, on the new nodes...

I would like to do this by tainting the original nodes so that rook-ceph is still happy, however when I applied the following it updated all csi pods, but not the mon or the osd:

placement:
  all:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: example.com/is-infra
            operator: In
            values:
            - "true"
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/infrastructure
      operator: Exists
  osd:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: example.com/is-infra
            operator: In
            values:
            - "true"
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/infrastructure
      operator: Exists

I did find this about the failover of the mon so I will give that a try, but idk if the same will work for osd? Thanks

[travisn]

The tolerations look correct to me. I would expect all the daemons including mons and OSDs to be updated with the tolerations. Any errors showing in the rook operator log? You aren't adding the taints to the nodes until after confirming the tolerations were added, right?

[a1994sc]

logs searching for osd or toleration https://gist.github.com/a1994sc/094bc4d300cb76aab4df09e9553ad8c9

I am waiting to taint the nodes until I see they have the tolerations:

kubectl -n rook-ceph get deploy rook-ceph-osd-0 -o yaml | yq .spec.template.spec.tolerations
null

kubectl -n rook-ceph get deploy rook-ceph-osd-0 -o yaml | yq .spec.template.spec.affinity
{}

[a1994sc]

Ok... so I have identified the problem and it is a bit of PEBKAC.... I had placement at the root of the values for the helm install, when it needs to be under cephClusterSpec

Here is the config that worked for me:

cephClusterSpec:
  placement:
    all:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: example.com/is-infra
              operator: In
              values:
              - "true"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infrastructure
        operator: Exists
    osd:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: example.com/is-infra
              operator: In
              values:
              - "true"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infrastructure
        operator: Exists
    mon:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: example.com/is-infra
              operator: In
              values:
              - "true"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infrastructure
        operator: Exists
    mgr:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
          - matchExpressions:
            - key: example.com/is-infra
              operator: In
              values:
              - "true"
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infrastructure
        operator: Exists

Going to close this as it has been answered