fix: Mark nodes that are pending deletion in cluster state for scheduling

[jonathan-innis]

Fixes #

Description

• Enables pods that are under Expiry with PDBs to schedule a node that will fit all the pods that are on a node marked for deletion rather than slowly trickling the pending pods onto a large number of small nodes

How was this change tested?

• make test
• Latency testing with large scale-ups
• TEST_FILTER=Integration make e2etests

Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: #
• No

Release Note

None

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✅ Deploy Preview for karpenter-docs-prod canceled.

Name	Link
🔨 Latest commit	6447a94
🔍 Latest deploy log	https://app.netlify.com/sites/karpenter-docs-prod/deploys/632007edd1e58d000947e1bf

[tzneal]

This didn't work for me when I pulled this branch and ran it. I have this pdb:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: inflate-pdb
  namespace: default
spec:
  maxUnavailable: 5
  selector:
    matchLabels:
      app: inflate

and I scaled up 100 inflate pods (the deployment is modified to use 250m CPU each) that landed on a single 8xlarge node and waited until they got running.

I deleted the node and we did launch an 8xlarge node at first, but there's some interaction with consolidation that isn't correct as when the node was half full, the new 8xlarge was replaced with a 4xlarge.

Original node w/100 pods - ip-192-168-173-189.us-west-2.compute.internal
Replacement node - ip-192-168-135-151.us-west-2.compute.internal

While the pods were slowly moving from the old node to the new node, consolidation tried to replace the new node with a smaller version:

NFO    controller.consolidation        Consolidating via Replace, terminating 1 nodes ip-192-168-135-151.us-west-2.compute.internal/c6g.8xlarge and replacing with a node from types r6g.4xlarge, r6gd.4xlarge, r5.4xlarge, vt1.3xlarge, c5a.4xlarge and 30 other(s)   {"commit": "59f4cf2"}

It looks like consolidation isn't counting the draining pods or it would have known that didn't work.

[jonathan-innis]

This didn't work for me when I pulled this branch and ran it. I have this pdb

Yep, that's a complete miss on my side. I updated the PR. Will add a test for your scenario as well

[tzneal]

lgtm