Microk8s comes with default containerd and its configurations could not find my Kata run time class #2110

santhoshcameo · 2021-06-24T14:25:49Z

santhoshcameo
Jun 24, 2021

I am trying Kata with Microkubernetes which is installed via sudo snap install microk8s --classic but microk8s comes with default containerd which causes configuration issues for me always. Please suggest any better alternatives.

I will share all the steps followed in this

   sudo apt-get update -y  
   sudo apt-get upgrade -y
   sudo snap install microk8s --classic
   sudo microk8s.status --wait-ready     
   sudo snap alias microk8s.kubectl kubectl   (
   sudo microk8s.enable dashboard dns
   sudo kubectl get pods --all-namespaces

Since Containerd comes with Microk8s by default, the following command will show the containerd status

sudo systemctl status snap.microk8s.daemon-containerd.service
from the above command , I could see that the configuration file location which containerd uses which will be something like /var/snap/microk8s/2264/args/containerd.toml

Note: Since containerd status shows the configuration path as/var/snap/microk8s/2264/args/containerd.toml I will add my Kata runtimes in this file.

santhosh_cameo@san-cd:~$ microk8s ctr -v
ctr github.com/containerd/containerd v1.3.7

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes]
        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata]
          runtime_type = "io.containerd.kata.v2"

Now I will install Kata

sudo snap install kata-containers --stable --classic

sudo mkdir -p /etc/kata-containers
sudo cp /snap/kata-containers/current/usr/share/defaults/kata-containers/configuration.toml /etc/kata-containers/
sudo cp /snap/kata-containers/<NUMBER>/usr/share/kata-containers/kata-containers.img /etc/kata-containers/

then make these changes to reflect in
/etc/kata-containers/configuration.toml

[hypervisor.qemu]
path = "/snap/kata-containers/current/usr/bin/qemu-system-x86_64
"kernel = "/snap/kata-containers/current/usr/share/kata-containers/vmlinux.container
"image = "/etc/kata-containers/kata-containers.img
"machine_type = "pc"

Now creating the symlink

sudo ln -sf /snap/kata-containers/current/usr/bin/containerd-shim-kata-v2 /usr/bin/containerd-shim-kata-v2
 
sudo ln -sf  /snap/kata-containers/current/usr/bin/kata-runtime /usr/bin/kata-runtime

Now I will create a Kata run time class kata-runtime.yaml

kind: RuntimeClass
  
apiVersion: node.k8s.io/v1beta1
  
metadata:
  
    name: kata-containers
  
handler: kata

then kubectl apply -f kata-runtime.yaml to create a Kata run time

After this I will have runtime class of Kata

santhosh_cameo@san-june-test:~$ kubectl get runtimeclass
NAME              HANDLER   AGE
kata-containers   kata      26h

Now restarting microk8s daemon to reflect everything

microk8s stop
 
microk8s start

Result:
By doing all the above mentioned steps, I will be able to run microk8s ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh with kata run time.

But if I run a pod with above created Kata runtime class I will end up with Error from server (Forbidden): error when creating "nginx-kata.yaml": pods "nginx-kata" is forbidden: pod rejected: RuntimeClass "kata" not found such errors.

apiVersion: v1
 
kind: Pod
 
metadata:
 
  name: nginx-kata
 
spec:
 
  runtimeClassName: kata
 
  containers:
 
  - name: nginx
 
    image: nginx

So please suggest, what is the right way or which steps I am missing in between. This happens with microk8s yaml deployment /pods only but as I mentioned the microk8s ctr run --runtime io.containerd.run.kata.v2 -t --rm docker.io/library/busybox:latest hello sh always works.

devimc · 2021-06-24T14:38:05Z

devimc
Jun 24, 2021
Maintainer

can you describe the runtimeclass (kubectl describe runtimeclass or microk8s kubectl describe runtimeclass )?

1 reply

santhoshcameo Jun 24, 2021
Author

@devimc here it is.

santhosh_cameo@san-june-test:~$ kubectl describe runtimeclass
Name:         kata-containers
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  node.k8s.io/v1
Handler:      kata
Kind:         RuntimeClass
Metadata:
  Creation Timestamp:  2021-06-23T12:33:49Z
  Managed Fields:
    API Version:  node.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:handler:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-06-23T12:33:49Z
  Resource Version:  4282
  Self Link:         /apis/node.k8s.io/v1/runtimeclasses/kata-containers
  UID:               362a2716-ac24-4c4d-81f7-2913228e5855
Events:              <none>
Name:         native
Namespace:    
Labels:       <none>
Annotations:  <none>
API Version:  node.k8s.io/v1
Handler:      runc
Kind:         RuntimeClass
Metadata:
  Creation Timestamp:  2021-06-23T12:34:37Z
  Managed Fields:
    API Version:  node.k8s.io/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:handler:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2021-06-23T12:34:37Z
  Resource Version:  4339
  Self Link:         /apis/node.k8s.io/v1/runtimeclasses/native
  UID:               17d7c77c-381a-4dae-8a2a-270c34d9a171
Events:              <none>
santhosh_cameo@san-june-test:~$

santhoshcameo · 2021-06-24T14:53:28Z

santhoshcameo
Jun 24, 2021
Author

The main issue is with containerd configuration file, and I am pretty sure about it too. it's tough to understand in which containerd configuration file we need to configure the kata run time,

1. should I dump the default to /etc/containerd/config.toml
/var/snap/microk8s/2262/args/containerd dump config > /etc/containerd/config.toml

then restart the microk8s cluster
or

Make the Kata changes in below-highlighted configuration which we get from sudo systemctl status snap.microk8s.daemon-containerd.service
```
    `├─24598 /snap/microk8s/2262/bin/containerd --config **/var/snap/microk8s/2262/args/containerd.toml**`
```

But both tried as well, and still could not detect the Kata run time :(

1 reply

devimc Jun 24, 2021
Maintainer

according to this https://microk8s.io/docs/configuring-services

snap.microk8s.daemon-containerd

Containerd is the container runtime used by MicroK8s to manage images and execute containers.

The containerd daemon started using the configuration in
${SNAP_DATA}/args/containerd and ${SNAP_DATA}/args/containerd-template.toml

maybe you are modifying the wrong file?

santhoshcameo · 2021-06-24T15:10:41Z

santhoshcameo
Jun 24, 2021
Author

thats the main issue here , but you can see from the status command, the last line which says about the snap location where the contained toml is loaded...here I have added the Kata. and restarted the daemon but no use.

santhosh_cameo@san-june-test:~$ sudo systemctl status snap.microk8s.daemon-containerd.service
● snap.microk8s.daemon-containerd.service - Service for snap application microk8s.daemon-containerd
   Loaded: loaded (/etc/systemd/system/snap.microk8s.daemon-containerd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-06-24 13:13:54 UTC; 1h 48min ago
 Main PID: 24598 (containerd)
    Tasks: 108 (limit: 4915)
   CGroup: /system.slice/snap.microk8s.daemon-containerd.service
           ├─24598 /snap/microk8s/2262/bin/containerd --config /var/snap/microk8s/2262/args/containerd.toml

1 reply

devimc Jun 24, 2021
Maintainer

yes, I guess /var/snap/microk8s/2262/args/containerd.toml is regenerated when the service is restarted, hence your changes are reverted, so maybe you should edit ${SNAP_DATA}/args/containerd-template.toml instead.. I'm just guessing since I don't have any experience with microk8s 🤔

santhoshcameo · 2021-06-24T15:23:26Z

santhoshcameo
Jun 24, 2021
Author

Basically the the snap has two folder under microk8s

santhosh_cameo@san-june-test:/var/snap/microk8s$ ls
2262  common  current
santhosh_cameo@san-june-test:/var/snap/microk8s$ ls current/args/
cluster-agent  containerd      containerd-template.toml  ctr   flannel-network-mgr-config  flanneld  kube-apiserver           kube-proxy      kubectl      kubelet
cni-network    containerd-env  containerd.toml           etcd  flannel-template.conflist   ha-conf   kube-controller-manager  kube-scheduler  kubectl-env  kubelite
santhosh_cameo@san-june-test:/var/snap/microk8s$ ls 2262/args/
cluster-agent  containerd      containerd-template.toml  ctr   flannel-network-mgr-config  flanneld  kube-apiserver           kube-proxy      kubectl      kubelet
cni-network    containerd-env  containerd.toml           etcd  flannel-template.conflist   ha-conf   kube-controller-manager  kube-scheduler  kubectl-env  kubelite
santhosh_cameo@san-june-test:/var/snap/microk8s$

I have added the kata templated in both folder 2262 and current containerd-template.toml files to test 😢

4 replies

devimc Jun 24, 2021
Maintainer

does it work?

santhoshcameo Jun 24, 2021
Author

No 😢

santhosh_cameo@san-june-test:~$ kubectl apply -f nginx-kata.yaml 
Error from server (Forbidden): error when creating "nginx-kata.yaml": pods "nginx-kata" is forbidden: pod rejected: RuntimeClass "kata" not found

devimc Jun 24, 2021
Maintainer

I'm running out of ideas, maybe this is a question for microk8s maintainers/experts ?

santhoshcameo Jun 24, 2021
Author

yes, posted a question over there, if the solution works I will write here, meanwhile if anyone else has some thoughts please help me.

eadamsintel · 2021-06-28T23:04:40Z

eadamsintel
Jun 28, 2021
Collaborator

I gave this a try because I wanted to try out microk8s. I ran into the same issue as you when following your instructions. I was able to get this to work with one small change. Change the metadata name from kata-containers to kata in the runtimeclass yaml so that it matches the handler. Hopefully this fixes it for you as well.

kind: RuntimeClass
apiVersion: node.k8s.io/v1beta1
metadata:
    name: kata
handler: kata

$ sudo microk8s.kubectl get pods
NAME         READY   STATUS    RESTARTS   AGE
nginx-kata   1/1     Running   0          2m40s

$ ps aux | grep qemu-system
root      208563  0.6  0.0 2533096 156348 ?      Sl   22:52   0:01 /snap/kata-containers/720/usr/bin/qemu-system-x86_64 -name sandbox-9c53ef00205aabc5ab68a2d140d87080980d46e8a407875083e9bda2a45c6391 -uuid 52d0a840-d6ba-4faf-a0e1-f99549779d54 -machine pc,accel=kvm,kernel_irqchip,nvdimm=on -cpu host,pmu=off -qmp unix:/run/vc/vm/9c53ef00205aabc5ab68a2d140d87080980d46e8a407875083e9bda2a45c6391/qmp.sock,server=on,wait=off -m 2048M,slots=10,maxmem=193140M -device pci-bridge,bus=pci.0,id=pci-bridge-0,chassis_nr=1,shpc=on,addr=2 -device virtio-serial-pci,disable-modern=false,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/vc/vm/9c53ef00205aabc5ab68a2d140d87080980d46e8a407875083e9bda2a45c6391/console.sock,server=on,wait=off -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/etc/kata-containers/kata-containers.img,size=134217728 -device virtio-scsi-pci,id=scsi0,disable-modern=false -object rng-random,id=rng0,filename=/dev/urandom -device virtio-rng-pci,rng=rng0 -device vhost-vsock-pci,disable-modern=false,vhostfd=3,id=vsock-3191745492,guest-cid=3191745492 -chardev socket,id=char-0ea6dcb49180a99c,path=/run/vc/vm/9c53ef00205aabc5ab68a2d140d87080980d46e8a407875083e9bda2a45c6391/vhost-fs.sock -device vhost-user-fs-pci,chardev=char-0ea6dcb49180a99c,tag=kataShared -netdev tap,id=network-0,vhost=on,vhostfds=4,fds=5 -device driver=virtio-net-pci,netdev=network-0,mac=d6:60:1a:6e:b7:ba,disable-modern=false,mq=on,vectors=4 -rtc base=utc,driftfix=slew,clock=host -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic --no-reboot -daemonize -object memory-backend-file,id=dimm1,size=2048M,mem-path=/dev/shm,share=on -numa node,memdev=dimm1 -kernel /snap/kata-containers/720/usr/share/kata-containers/vmlinux-5.10.25.container -append tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k console=hvc0 console=hvc1 cryptomgr.notests net.ifnames=0 pci=lastbus=0 root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro ro rootfstype=ext4 quiet systemd.show_status=false panic=1 nr_cpus=96 systemd.unit=kata-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket scsi_mod.scan=none -pidfile /run/vc/vm/9c53ef00205aabc5ab68a2d140d87080980d46e8a407875083e9bda2a45c6391/pid -smp 1,cores=1,threads=1,sockets=96,maxcpus=96
eadams    216293  0.0  0.0   5192  2496 pts/1    S+   22:56   0:00 grep --color=auto qemu-system

1 reply

santhoshcameo Jun 29, 2021
Author

@eadamsintel Yes Eric, that was the cause and the issue is fixed. I was using the handler throughout my pod declaration instead as you suggested used the name in pod declaration. Thanks a million.
But I guess this thread/steps will help newbies/anyone to start with Mikcrok8s-contained with Kata with very minimal steps.

Thanks again @devimc @eadamsintel

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Microk8s comes with default containerd and its configurations could not find my Kata run time class #2110

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 5 comments 8 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Microk8s comes with default containerd and its configurations could not find my Kata run time class #2110

santhoshcameo Jun 24, 2021

Replies: 5 comments · 8 replies

devimc Jun 24, 2021 Maintainer

santhoshcameo Jun 24, 2021 Author

santhoshcameo Jun 24, 2021 Author

devimc Jun 24, 2021 Maintainer

santhoshcameo Jun 24, 2021 Author

devimc Jun 24, 2021 Maintainer

santhoshcameo Jun 24, 2021 Author

devimc Jun 24, 2021 Maintainer

santhoshcameo Jun 24, 2021 Author

devimc Jun 24, 2021 Maintainer

santhoshcameo Jun 24, 2021 Author

eadamsintel Jun 28, 2021 Collaborator

santhoshcameo Jun 29, 2021 Author

santhoshcameo
Jun 24, 2021

Replies: 5 comments 8 replies

devimc
Jun 24, 2021
Maintainer

santhoshcameo Jun 24, 2021
Author

santhoshcameo
Jun 24, 2021
Author

devimc Jun 24, 2021
Maintainer

santhoshcameo
Jun 24, 2021
Author

devimc Jun 24, 2021
Maintainer

santhoshcameo
Jun 24, 2021
Author

devimc Jun 24, 2021
Maintainer

santhoshcameo Jun 24, 2021
Author

devimc Jun 24, 2021
Maintainer

santhoshcameo Jun 24, 2021
Author

eadamsintel
Jun 28, 2021
Collaborator

santhoshcameo Jun 29, 2021
Author