Helm deployment with multiple worker configuration profiles

[ghost]

In an effort to provide cost-optimization on the cluster scaling, I've been trying to schedule worker pods on to different sized nodes. I have two nodepools: small and large. I set the default pod limits to 1vcpu and 11GB RAM (wrt RAM, two fit on a small node and four would fit on to a large node), however, I also want some users to, through the cluster.options configuration when starting a cluster, to request higher limits (3vcpus and 21GB RAM for a full small node or even 7vcpu and 45GB RAM for a large node).

Below is the NodeAffinity solution that kind of works, however, when a large enough cluster is scaled up with the default limits (1vcpu and 11GB RAM), sometimes this triggers the scale up of the larger nodes. This isn't a surprise due to the kube-scheduler and the non-guarantee of the preferredDuringSchedulingIgnoredDuringExecution preferences.

dask-gateway:
 gateway:
  baskend:
   worker:
        extraPodConfig:
          tolerations:
            - key: node_toleration
              value: worker
              operator: Equal
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node_label
                    operator: In
                    values:
                      - worker
                      - lgworker
              preferredDuringSchedulingIgnoredDuringExecution:
                - weight: 100
                  preference:
                    matchExpressions:
                    - key: node_label
                      operator: In
                      values: 
                      - worker
                - weight: 1
                  preference:
                    matchExpressions:
                    - key: node_label
                      operator: In
                      values: 
                      - lgworker

So, I was wondering, is it feasible to have multiple profiles for worker pods? Something that would effectively extend the cluster profile options here.

Otherwise, perhaps someone may have some better insight in how to achieve what I'm trying to accomplish.

[ghost]

dask/dask-kubernetes#384

dask/helm-chart#221