Deb package upgrade from 1.29.1-1.1 to 1.29.2-1.1 moves conmon/crun

[jafager]

What happened?

Updated cri-o package on Ubuntu 22.04 from 1.29.1-1.1 to 1.29.2-1.1 on Kubernetes 1.29 nodes. Nodes subsequently could not create containers (CreateContainerError) as /usr/bin/conmon and /usr/bin/crun had been renamed to /usr/bin/crio-conmon and /usr/bin/crio-crun (looking at past issues it seems like this was done to resolve conflicts in the Red Hat ecosystem?). Rolling back the package appears to have fixed this problem temporarily.

What did you expect to happen?

I did not see anything to indicate that 1.29.1 to 1.29.2 would be a breaking change so I did not expect to see significant impact.

How can we reproduce it (as minimally and precisely as possible)?

Install cri-o=1.29.1-1.1 on Ubuntu 22.04 with Kubernetes 1.29, upgrade to cri-o=1.29.2-1.1, attempt to start containers, should see ContainerCreateError with an event complaining that /usr/bin/conman is not found.

Anything else we need to know?

I'm not sure if I missed something in the upgrade process, or if there's something wrong with the package (e.g. files got moved but cri-o was still looking for them in the wrong place).

CRI-O and Kubernetes version

$ crio --version
crio version 1.29.1
Version:        1.29.1
GitCommit:      78e179ba8dd3ce462382a17049e8d1f770246af1
GitCommitDate:  2024-01-11T21:36:06Z
GitTreeState:   clean
BuildDate:      1970-01-01T00:00:00Z
GoVersion:      go1.21.1
Compiler:       gc
Platform:       linux/amd64
Linkmode:       static
BuildTags:
  static
  netgo
  osusergo
  exclude_graphdriver_btrfs
  exclude_graphdriver_devicemapper
  seccomp
  apparmor
  selinux
LDFlags:          unknown
SeccompEnabled:   true
AppArmorEnabled:  false

$ kubectl version --output=json
{
  "clientVersion": {
    "major": "1",
    "minor": "29",
    "gitVersion": "v1.29.1",
    "gitCommit": "bc401b91f2782410b3fb3f9acf43a995c4de90d2",
    "gitTreeState": "clean",
    "buildDate": "2024-01-17T15:51:03Z",
    "goVersion": "go1.21.6",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "kustomizeVersion": "v5.0.4-0.20230601165947-6ce0bf390ce3"
}

OS version

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.4 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy
$ uname -a
Linux REDACTED 5.15.0-101-generic #111-Ubuntu SMP Tue Mar 5 20:16:58 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Additional environment details (AWS, VirtualBox, physical, etc.)

VMware ESXi

[github-actions]

A friendly reminder that this issue had no activity for 30 days.