EC2 Nodeclass using Bottlerocket image won't provision larger root volume

[seanpascual]

Description

Observed Behavior:
We are running Karpenter with AWS Bottlerocket nodes, but have seen that since moving to Bottlerocket AMIs from AL2 that the /dev/root volume is always >80% full.

The documentation states that a 4Gi EBS volume should be provisioned by default for /dev/root and a 20Gi EBS volume for data.

We are seeing, however, that only a 1Gi /dev/root volume is provisioned.

Doc link: https://karpenter.sh/v0.34/concepts/nodeclasses/#bottlerocket-1

We have therefore tried overriding the blockDeviceMapping both for root and data. This appears to work for the data volume, but not for root.

Checking mounted volume sizes on the instance one can see that /dev/nvme1n1p1 is at the expected 50Gi, however the /dev/root volume is still at 1Gi.

This only appears to be happening with Bottlerocket nodes as it is not affecting our AL2 nodes / EC2NodeClass.

Expected Behavior:

We are expecting that the root volume should be overridden to 5Gi and the data volume be overridden to 50Gi.

Reproduction Steps (Please include YAML):

Provisioning the EC2NodeClass:

defaults:
  ec2NodeClass:
    spec:
      securityGroupSelectorTerms:
      - tags:
          Name: xxx-node
      tags:
        # Matches the tag in the IRSA policy
        karpenter.k8s.aws/cluster: xxx
      blockDeviceMappings:
        - deviceName: /dev/xvda
          ebs:
            volumeSize: 5Gi
            volumeType: gp3
            encrypted: true
            deleteOnTermination: true
        - deviceName: /dev/xvdb
          ebs:
            volumeSize: 50Gi
            volumeType: gp3
            iops: 3000
            encrypted: true
            deleteOnTermination: true
            throughput: 125

Describing the EC2NodeClass after deploying the above config:

> k describe ec2nodeclass default
Name:         default
Namespace:    
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  karpenter.k8s.aws/ec2nodeclass-hash: xxx
              karpenter.k8s.aws/ec2nodeclass-hash-version: v2
              meta.helm.sh/release-name: karpenter-provisioning
              meta.helm.sh/release-namespace: karpenter
API Version:  karpenter.k8s.aws/v1beta1
Kind:         EC2NodeClass
Metadata:
  Creation Timestamp:  2024-02-27T11:57:06Z
  Finalizers:
    karpenter.k8s.aws/termination
  Generation:        6
  Resource Version:  748924897
  UID:               22b5f79e-a4aa-4903-8b9f-4aabc1d4c012
Spec:
  Ami Family:  Bottlerocket
  Block Device Mappings:
    Device Name:  /dev/xvda
    Ebs:
      Delete On Termination:  true
      Encrypted:              true
      Volume Size:            5Gi
      Volume Type:            gp3
    Device Name:              /dev/xvdb
    Ebs:
      Delete On Termination:  true
      Encrypted:              true
      Iops:                   3000
      Throughput:             125
      Volume Size:            50Gi
      Volume Type:            gp3

Checking the volume sizes on the instance shows that the /dev/root volume hasn't increased in size, while the "data" volume has increased as expected:

/dev/nvme1n1p1   50G  7.5G   43G  15% /.bottlerocket/support
/dev/root       904M  676M  166M  81% /usr/local/bin/apiclient

Versions:

• Chart Version: 0.36.1
• Kubernetes Version (kubectl version): v1.29.3-eks-adc7111

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[jmdeal]

I was able to reproduce, and it does look like Karpenter is provisioning the requested EBS volumes but that is not reflected in the root partition. I'm not a bottlerocket expert, but my understanding is that the root partition is immutable so it being a fixed size makes sense.

[ssm-user@control]$ df -h
Filesystem      Size  Used Avail Use% Mounted on
overlay          50G  3.6G   47G   8% /
tmpfs            64M     0   64M   0% /dev
shm              64M     0   64M   0% /dev/shm
tmpfs            64M     0   64M   0% /run
tmpfs           3.8G  504K  3.8G   1% /etc/hosts
tmpfs           1.6G  1.5M  1.6G   1% /run/api.sock
/dev/nvme1n1p1   50G  3.6G   47G   8% /.bottlerocket/support
/dev/root       904M  592M  250M  71% /usr/local/bin/apiclient
tmpfs           3.8G     0  3.8G   0% /proc/acpi
tmpfs           3.8G     0  3.8G   0% /proc/scsi

You can see that when I list the block devices from the admin container that the 5gb root volume is present.

[root@admin]# lsblk
NAME         MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
loop0          7:0    0 12.4M  1 loop /.bottlerocket/rootfs/var/lib/kernel-devel/.overlay/lower
loop1          7:1    0  308K  1 loop /.bottlerocket/rootfs/aarch64-bottlerocket-linux-gnu/sys-root/usr/share/licenses
nvme0n1      259:0    0    5G  0 disk
|-nvme0n1p1  259:3    0    4M  0 part
|-nvme0n1p2  259:4    0    5M  0 part
|-nvme0n1p3  259:5    0   40M  0 part /.bottlerocket/rootfs/boot
|-nvme0n1p4  259:6    0  920M  0 part
|-nvme0n1p5  259:7    0   10M  0 part
|-nvme0n1p6  259:8    0   25M  0 part
|-nvme0n1p7  259:9    0    5M  0 part
|-nvme0n1p8  259:10   0   40M  0 part
|-nvme0n1p9  259:11   0  920M  0 part
|-nvme0n1p10 259:12   0   10M  0 part
|-nvme0n1p11 259:13   0   25M  0 part
|-nvme0n1p12 259:14   0   41M  0 part /.bottlerocket/rootfs/var/lib/bottlerocket
`-nvme0n1p13 259:15   0    1M  0 part
nvme1n1      259:1    0   50G  0 disk
`-nvme1n1p1  259:16   0   50G  0 part /.bottlerocket/rootfs/local

What is your goal in increasing the size of the root partition? I do think this is likely more a bottlerocket question than a Karpenter question.

[seanpascual]

Hi @jmdeal Thanks for looking into this. We were looking to increase the size of the root volume solely because it kept on triggering disk space low alerts in our monitoring systems.

Since the Karpenter documentation mentioned that a custom disk size could be set, we thought that it would be an easy way to give the disk a bit more breathing room.

If the disk size is actually immutable though, we will probably just silence the alerts.

[github-actions]

This issue has been inactive for 14 days. StaleBot will close this stale issue after 14 more days of inactivity.