Failing to provision EC2 node even though matching EC2 node type exists

[kamialie]

Description

Observed Behavior:

Karpenter can not schedule the pod claiming that scheduling requirements were not met.

The pod should run on i3en.24xlarge EC2 node, and as far as I can tell the resource requests satisfy the resources available on this node type (including daemonset pods overhead). If i3en.24xlarge node is already running in the cluster Kubernetes scheduler successfully assigns this pod to run on that node, the only problem is to get Karpenter to provision it. One way I was able to do that is to scale up another deployment, where CPU requests is set to 40, thus, 2 Pods force Karpenter to provision the node I need, so that Kubernetes scheduler can then assign my initial pod to it.

Setting Karpenter log level to debug unfortunately didn't reveal anything at all (I have just set logLevel Helm chart property to debug, is there some other place to increase it even more?). Here is what Karpenter says at the moment:

incompatible with nodepool \"example\", daemonset overhead={\"cpu\":\"275m\",\"memory\":\"360Mi\",\"pods\":\"7\"}, no instance type satisfied resources {\"cpu\":\"80275m\",\"memory\":\"737640Mi\",\"pods\":\"8\"} and requirements karpenter.k8s.aws/instance-family In [i3en], karpenter.k8s.aws/instance-size NotIn [metal], karpenter.sh/capacity-type In [on-demand], karpenter.sh/nodepool In [example], kubernetes.io/arch In [amd64], kubernetes.io/os In [linux], role In [example], topology.kubernetes.io/zone In [eu-central-1a eu-central-1b eu-central-1c] (no instance type which had enough resources and the required offering met the scheduling requirements)

Overall it looks like that something is wrong with how Karpenter calculates memory requirements, or maybe the problem is on AWS side - whatever it returns to Karpenter when it tries to fetch information on available node types, thus, the issue could be related to Karpenter or AWS provider. Maybe the reason lies in the fact that when such node is running, Kubernetes reports that it consumes 97%, and there is some kind of a threshold that Karpenter doesn't want to break? I would still expect Karpenter to be able to provision a node, since scheduler didn't have problems to assing that pod.

Expected Behavior:

Karpenter identifies matching node and provisions it.

Reproduction Steps (Please include YAML):

Karpenter node definition:

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: example
spec:
  amiFamily: AL2
  instanceProfile: KarpenterNode
  securityGroupSelectorTerms:
  - tags:
      karpenter.sh/discovery: dev
  subnetSelectorTerms:
  - tags:
      karpenter.sh/discovery: dev
---
apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: example
spec:
  disruption:
    consolidateAfter: 15m
    consolidationPolicy: WhenEmpty
    expireAfter: 336h
  template:
    metadata:
      labels:
        role: example
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: example
      requirements:
      - key: kubernetes.io/arch
        operator: In
        values:
        - amd64
      - key: kubernetes.io/os
        operator: In
        values:
        - linux
      - key: topology.kubernetes.io/zone
        operator: In
        values:
        - eu-central-1a
        - eu-central-1b
        - eu-central-1c
      - key: karpenter.sh/capacity-type
        operator: In
        values:
        - on-demand
      - key: karpenter.k8s.aws/instance-family
        operator: In
        values:
        - i3en
      - key: karpenter.k8s.aws/instance-size
        operator: NotIn
        values:
        - metal

Sample pod:

apiVersion: v1
kind: Pod
metadata:
  name: example
  namespace: default
spec:
  nodeSelector:
    role: example
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: nginx
    resources:
      requests:
        cpu: "80"
        memory: 720Gi

Versions:

• Chart Version: 0.36.0
• Kubernetes Version (kubectl version):

Client Version: v1.29.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.3-eks-adc7111

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[thehandsomezebra]

I am currently experiencing the exact same issue - currently attempting to troubleshoot this with AWS now. We are running chart version 0.35.0

Edit: We are using amiFamily: Bottlerocket && Spot + On Demand instances.

[kamialie]

Right, I didn't mention that our setup just recently upgraded from k8s 1.27 and Karpenter 0.28.1, so this exists for quite some time.

[engedaam]

You might need to adjust the VM_MEMORY_OVERHEAD_PERCENT, currently karpenter will take a set percent of the memory off the top for all instance types of 7.5% https://karpenter.sh/docs/reference/settings/. On the EC2 side there is an PR to dynamically add the memory overhead more dynamically for each instance type. aws/karpenter-provider-aws#4517. I recommended moving this issue the AWS provider repo https://github.com/aws/karpenter-provider-aws. Also, this issue might also help better help in scoping this behavior down aws/karpenter-provider-aws#5161

[kamialie]

Thanks a lot for proving so much context and related issues/pull requests! I'll double check what kubelet reports for this instance type and see if math makes sense with Karpenter added overhead, but that seems to explain it.

The core problem of memory being reported less by kubelet may have different solutions, I could also think of an optional field in ec2nodeclass, but of course I'm not aware how majority cuts these classes for their use cases and if this field would be of help. However, would definitely help my use case.

[engedaam]

/remove-kind bug
/remove needs-triage
/kind support

[engedaam]

/triage accepted

[jcmcken]

Also seeing this issue in 0.36.1. We're trying to use GPU nodes and setting nodeSelector to karpenter.k8s.aws/instance-category: g. If a node with that category already exists in the cluster, it works. If not, I get the same errors as OP.

I even checked my node pool requirements settings against https://karpenter.sh/docs/reference/instance-types/#g5xlarge (for example), and they should match and use this instance type. But it doesn't seem to work

[jcmcken]

Also seeing this issue in 0.36.1. We're trying to use GPU nodes and setting nodeSelector to karpenter.k8s.aws/instance-category: g. If a node with that category already exists in the cluster, it works. If not, I get the same errors as OP.

I even checked my node pool requirements settings against https://karpenter.sh/docs/reference/instance-types/#g5xlarge (for example), and they should match and use this instance type. But it doesn't seem to work

Nevermind, I think the problem is AWS was out of all the instance types that our node pool was selecting. When I broadened the node pool requirements it was able to find a match finally. Also enabled the Prometheus metrics which have some useful metrics on instance type availability. Maybe the error message in this instance is a little bit too vague: no instance type which had enough resources and the required offering met the scheduling requirements