Guest VMs are not up, when Host VM OS is ubuntu 22.04 and kernel 5.15

[scydas]

What happened:
A clear and concise description of what the bug is.
The Host OS ubuntu 22.04 and kernel 5.15.0-57-generic , We are seeing the Kubevirt images not up properly, due to which we are not able to login to the VMs.

vnc error:

virt-launcher log:

virt-handler log:

What you expected to happen:
A clear and concise description of what you expected to happen.
The guest VMs should be running and user should be able to get into the guest VMs without any issues.

How to reproduce it (as minimally and precisely as possible):
Create Ubuntu 22.04 VM with kernel 5.15.0-57-generic,(issue seen on kernels 5.8+).
Install k8s, kubevirt.
Create VMs using kubevirt on the Ubuntu 22.04 VMs with kernel 5.15.

Additional context:
Add any other context about the problem here.
use https://kubevirt.io/labs/manifests/vm.yaml creat vm also error

Environment:

• KubeVirt version (use virtctl version): 1.0
• Kubernetes version (use kubectl version): v1.28.6
• VM or VMI specifications: N/A
• Cloud provider or hardware configuration: N/A
• OS (e.g. from /etc/os-release): Ubuntu 22.04（Host）
• Kernel (e.g. uname -a): 5.15.0-78-generic
• Install tools: operator
• Others: N/A

[alicefr]

Can you please provide us the VM yaml you used for the first VM

[alicefr]

I would try to increase the memory given to the guest, just as an easy check. Then, you should try to run virt-host-validate to check if there are any issues on your nodes (see the kubevirt documentation)

[victortoso]

Just to be sure, your node OS is ubuntu 22.04 with kernel 5.15* and with that, you see crashes in both ubuntu 22.04 VMs and also cirros (kernel 4.4.0?"

#edit (sent by mistake)

I'm wondering if this is not a kernel bug. I see at least one fix after 5.15 that might be related (cpu stuck). If this is indeed a kernel bug on your node OS, you should report it to them for proper tracking/fixing.

[scydas]

Can you please provide us the VM yaml you used for the first VM

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: test2
  namespace: default
spec:
  dataVolumeTemplates:
  - metadata:
      name: test2-rootdisk
      namespace: default
    spec:
      pvc:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
        storageClassName: local-path
      source:
        registry:
          url: docker://system-images/ubuntu-22.04-x86_64:v1
  runStrategy: Always
  template:
    metadata:
      creationTimestamp: null
    spec:
      architecture: amd64
      domain:
        cpu:
          cores: 1
          sockets: 1
          threads: 1
        devices:
          disks:
          - bootOrder: 1
            disk:
              bus: virtio
            name: rootdisk
          - disk:
              bus: virtio
            name: cloudinitdisk
          interfaces:
          - masquerade: {}
            name: default
        machine:
          type: q35
        resources:
          requests:
            memory: 2Gi
      networks:
      - name: default
        pod: {}
      volumes:
      - dataVolume:
          name: test2-rootdisk
        name: rootdisk
      - cloudInitNoCloud:
          userDataBase64: I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlCmRpc2FibGVfcm9vdDogZmFsc2UKY2hwYXNzd2Q6IHsibGlzdCI6ICJyb290OmRhbmdlcm91cyIsIGV4cGlyZTogRmFsc2V9CgoKcnVuY21kOgogIC0gc2VkIC1pICIvI1w/UGVybWl0Um9vdExvZ2luL3MvXi4qJC9QZXJtaXRSb290TG9naW4geWVzL2ciIC9ldGMvc3NoL3NzaGRfY29uZmlnCiAgLSBzeXN0ZW1jdGwgcmVzdGFydCBzc2guc2VydmljZQogIC0gZGhjbGllbnQgLXIgJiYgZGhjbGllbnQ=
        name: cloudinitdisk

system image dockerfile:

FROM scratch
ADD --chown=107:107 ubuntu-22.04-x86_64.img /disk/

os img :
https://cloud-images.ubuntu.com/jammy/20230828/jammy-server-cloudimg-amd64.img

[scydas]

I would try to increase the memory given to the guest, just as an easy check. Then, you should try to run virt-host-validate to check if there are any issues on your nodes (see the kubevirt documentation)

[scydas]

Just to be sure, your node OS is ubuntu 22.04 with kernel 5.15* and with that, you see crashes in both ubuntu 22.04 VMs and also cirros (kernel 4.4.0?"

#edit (sent by mistake)

I'm wondering if this is not a kernel bug. I see at least one fix after 5.15 that might be related (cpu stuck). If this is indeed a kernel bug on your node OS, you should report it to them for proper tracking/fixing.

node os is ubuntu 22.0.4 with 5.15*, guest os is ubuntu 22.04(Linux test2 5.15.0-76-generic #83-Ubuntu SMP Thu Jun 15 19:16:32 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux) and cirros(Linux testvm 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 GNU/Linux)

[alicefr]

I agree with Victor, the issue seems more related to your host. Any chance to try to boot a VM on the same OS without k8s and kubevirt?

[scydas]

I agree with Victor, the issue seems more related to your host. Any chance to try to boot a VM on the same OS without k8s and kubevirt?

create a vm by kvm or qemu ？

[victortoso]

create a vm by kvm or qemu ?

If you can try to create similar VM in that Node, it'll be easier to validate the bug. You can use libvirt + qemu for that.