Karpenter won't use fallback nodePool on insufficient instance capacity

[raychinov]

Description

Observed Behavior:
We have a default nodePool in one availability zone (a) with higher weight and a second fallback nodePool in another availability zone (b) with lower weight. The scenario we test is with AWS Fault Injection Simulator, where all a AZ instances are terminated and new instance launches are paused.
What we observe is new nodeClaims being created but stay in Non-Ready state and nodes fail to launch with an error message like:

creating instance, getting launch template configs, getting launch templates, no instance types satisfy requirements of amis ami-02f420afc14289ede

The concerning thing here is that Karpenter won't try to launch instances from the fallback nodePool even after several minutes of waiting.
The messages in the log are mostly:

{"level":"DEBUG","time":"2024-05-08T11:26:25.025Z","logger":"controller.disruption","message":"waiting on cluster sync","commit":"2c8f2a5"}
{"level":"DEBUG","time":"2024-05-08T11:26:26.026Z","logger":"controller.disruption","message":"waiting on cluster sync","commit":"2c8f2a5"}
{"level":"DEBUG","time":"2024-05-08T11:26:27.027Z","logger":"controller.disruption","message":"waiting on cluster sync","commit":"2c8f2a5"}

If we increase the fallback nodePool weight and bump the workload so new nodeClaims are created, they would start in the b AZ and get provisioned successfully, but the a AZ nodeClaims will stay pending as well as the pods meant to start on them.

Expected Behavior:
Unsuccessful nodeClaims would time out after a couple of seconds, and new ones will be created in an alternative nodePool. Also, having more verbose log messages would be appreciated.

Reproduction Steps (Please include YAML):

NodePools

---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
  annotations:
    kubernetes.io/description: "Default NodePool"
spec:
  weight: 100
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["c7g.large", "c7g.xlarge"]
        - key: topology.kubernetes.io/zone
          operator: In
          values:
            - "eu-west-1a"
      nodeClassRef:
        name: default
      taints:
        - key: karpenter
          value: "true"
          effect: NoSchedule
      kubelet:
        maxPods: 125
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: fallback
  annotations:
    kubernetes.io/description: "Fallback NodePool"
spec:
  weight: 10
  template:
    spec:
      requirements:
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot"]
        - key: node.kubernetes.io/instance-type
          operator: In
          values: ["c7g.large", "c7g.xlarge"]
        - key: topology.kubernetes.io/zone
          operator: In
          values:
            - "eu-west-1b"
            - "eu-west-1c"
      nodeClassRef:
        name: default
      taints:
        - key: karpenter
          value: "true"
          effect: NoSchedule
      kubelet:
        maxPods: 125

Workload

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test
spec:
  replicas: 2
  selector:
    matchLabels:
      app: test
  template:
    metadata:
      labels:
        app: test
    spec:
      securityContext:
        runAsUser: 65534
        fsGroup: 65534
      containers:
        - image: public.ecr.aws/eks-distro/kubernetes/pause:3.2
          name: test
          resources:
            requests:
              cpu: 50m
              memory: 64M
      nodeSelector:
        karpenter.k8s.aws/instance-hypervisor: nitro
      tolerations:
        - key: karpenter
          operator: Equal
          value: "true"
          effect: NoSchedule
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - test
              topologyKey: kubernetes.io/hostname

FIS template

{
    "description": "AZ fail",
    "targets": {
        "Karp-EC2-Instances": {
            "resourceType": "aws:ec2:instance",
            "resourceTags": {
                "karpenter.k8s.aws/ec2nodeclass": "default"
            },
            "filters": [
                {
                    "path": "State.Name",
                    "values": [
                        "running"
                    ]
                },
                {
                    "path": "Placement.AvailabilityZone",
                    "values": [
                        "eu-west-1a"
                    ]
                }
            ],
            "selectionMode": "ALL"
        },
        "Karp-IAM-roles": {
            "resourceType": "aws:iam:role",
            "resourceArns": [
                "arn:aws:iam::111111111111:role/karpenter-controller-role",
                "arn:aws:iam::111111111111:role/karpenter-node-role"
            ],
            "selectionMode": "ALL"
        }
    },
    "actions": {
        "Pause-Instance-Launches": {
            "actionId": "aws:ec2:api-insufficient-instance-capacity-error",
            "parameters": {
                "availabilityZoneIdentifiers": "eu-west-1a",
                "duration": "PT10M",
                "percentage": "100"
            },
            "targets": {
                "Roles": "Karp-IAM-roles"
            }
        },
        "Terminate-Instances": {
            "actionId": "aws:ec2:terminate-instances",
            "parameters": {},
            "targets": {
                "Instances": "Karp-EC2-Instances"
            }
        }
    },
    "stopConditions": [
        {
            "source": "none"
        }
    ],
    "roleArn": "arn:aws:iam::111111111111:role/service-role/AWSFISIAMRole-1714174839723",
    "tags": {
        "Name": "AZ Availability: insufficient-instance-capacity-error"
    },
    "experimentOptions": {
        "accountTargeting": "single-account",
        "emptyTargetResolutionMode": "skip"
    }
}

Versions:

• Chart Version: 0.35.0
• Kubernetes Version (kubectl version): 1.22.17

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jonathan-innis]

Can you share the NodePool and EC2NodeClass that you are using here? Can you also share the entire set of Karpenter controller logs from the FIS simulation? Can you also share what exactly you are doing/executing during the FIS simulation (it seems like you are just ICE-ing all instance types across the single AZ)?

[raychinov]

Hey Jonathan, thank you for looking into this. I've shared the NodePools definition in the issue description, and here are the EC2NodeClass and Controller logs requested. And yes, in the FIS simulation, we are terminating the Kapenter-managed nodes and ICE-ing all instance types across the eu-west-1a AZ. Also, we use a custom AMI in the EC2NodeClass, but I don't think this is what causes the issue.

Looking into the logs again, it seems to me that happens is:

• During the FIS simulation, we terminate instances in eu-west-1a and tell the EC2 API to start responding with api-insufficient-instance-capacity-error for eu-west-1a requests for some time
• Karpenter asks the API for new nodes in the eu-west-1a AZ to host the pending pods, but the requests get rejected and Karpenter removes the default NodePool instance types (just c7g.xlarge in this case, c7g.large seems to not fit the requirement) from the offerings:

{"level":"DEBUG","time":"2024-05-10T09:17:20.363Z","logger":"controller.nodeclaim.lifecycle","message":"removing offering from offerings","commit":"2c8f2a5","nodeclaim":"default-g6c6x","reason":"InsufficientInstanceCapacity","instance-type":"c7g.xlarge","zone":"eu-west-1a","capacity-type":"spot","ttl":"3m0s"}
{"level":"ERROR","time":"2024-05-10T09:17:20.363Z","logger":"controller.nodeclaim.lifecycle","message":"creating instance, insufficient capacity, with fleet error(s), InsufficientInstanceCapacity: We currently do not have sufficient capacity in the Availability Zone you requested.","commit":"2c8f2a5","nodeclaim":"default-g6c6x"}

• The thing is that the offerings seems to be removed not just for the eu-west-1a availability zone, but for the whole region offerings. Resulting in nodes not launched in the fallback NodePool/AZ

{"level":"ERROR","time":"2024-05-10T09:17:29.334Z","logger":"controller","message":"Reconciler error","commit":"2c8f2a5","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-6t977"},"namespace":"","name":"default-6t977","reconcileID":"ea21ad18-3206-49f1-a98b-2b19726ae3d5","error":"launching nodeclaim, creating instance, getting launch template configs, getting launch templates, no instance types satisfy requirements of amis ami-02fc209fc12289dde"}
{"level":"DEBUG","time":"2024-05-10T09:17:29.682Z","logger":"controller.disruption","message":"waiting on cluster sync","commit":"2c8f2a5"}
{"level":"DEBUG","time":"2024-05-10T09:17:29.788Z","logger":"controller.provisioner","message":"waiting on cluster sync","commit":"2c8f2a5"}

When we do the same tests with fallback NodePool containing a different set of instance types the fallback do work and new instances in eu-west-1b and eu-west-1c are provisined successfully.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.