I found that the issue is that the cgroup is in threaded mode, and in that case, reading cgroup.procs returns ENOTSUP.

By patching runc with the following patch, tests work again and runc doesn't fail:

diff --git a/libcontainer/cgroups/utils.go b/libcontainer/cgroups/utils.go
index b32af4ee..70080efd 100644
--- a/libcontainer/cgroups/utils.go
+++ b/libcontainer/cgroups/utils.go
@@ -19,6 +19,7 @@ import (
 
 const (
        CgroupProcesses   = "cgroup.procs"
+       CgroupThreads     = "cgroup.threads"
        unifiedMountpoint = "/sys/fs/cgroup"
        hybridMountpoint  = "/sys/fs/cgroup/unified"
 )
@@ -137,14 +138,16 @@ func GetAllSubsystems() ([]string, error) {
 }
 
 func readProcsFile(dir string) ([]int, error) {
-       f, err := OpenFile(dir, CgroupProcesses, os.O_RDONLY)
+       contents, err := ReadFile(dir, CgroupProcesses)
+       if errors.Is(err, unix.ENOTSUP) {
+               contents, err = ReadFile(dir, CgroupThreads)
+       }
        if err != nil {
                return nil, err
        }
-       defer f.Close()
 
        var (
-               s   = bufio.NewScanner(f)
+               s   = bufio.NewScanner(strings.NewReader(contents))
                out = []int{}
        )

Here is the type of the cgroups (these commands were run inside the buildkit's dev-env container:

# cat /sys/fs/cgroup/buildkit/mxv4shz9kwdm0p5u49mw971ft/cgroup.type
threaded
# cat /sys/fs/cgroup/buildkit/cgroup.type
threaded
# cat /sys/fs/cgroup/cgroup.type
domain threaded

Hi,
I have the same issue (with runc 1.1.10).
Having this patch applied to the next version would be awesome!

@Bacto we've changed this part of runc a lot in the main branch. Can you try to repro this using runc compiled from the main branch?

Hi @kolyshkin,

I tried with the main branch and got the same issue:

# runc -v
runc version 1.1.0+dev
commit: 0c5a735
spec: 1.1.0+dev
go: go1.21.6
libseccomp: 2.5.5

Here is the type of the cgroups (these commands were run inside the buildkit's dev-env container:

# cat /sys/fs/cgroup/buildkit/mxv4shz9kwdm0p5u49mw971ft/cgroup.type
threaded
# cat /sys/fs/cgroup/buildkit/cgroup.type
threaded
# cat /sys/fs/cgroup/cgroup.type
domain threaded

So the problem here is threaded cgroup type. In this case, processes actually belong to the cgroup parent which has "domain threaded" type (i.e. top cgroup in this case). It would be incorrect to send SIGKILL to specific threads in this group. So, basically, runc kill does the right thing here returning an error.

This is some kind of a misconfiguration, possibly caused by buildkit.

Created Debian 12 VM, checked in buildkit and ran its test suite inside a container (make test). Was not able to reproduce.

I think there was something wrong originally when starting a container.

Would still like to get to the bottom of it, so any suggestions of how to reproduce it (ideally a vagrant file or something like this) are welcome.

The issue is fixed in main branch.
I've tried again the 1.1.5 version and reproduced it, but I don't reproduce it with the main branch of runc.

I've tried to find the first fixed version and found that I can reproduce the same issue with 1.1.12 but not anymore with 1.2.0-rc.1.

So I'm closing this issue.

For reference, I'm using go test -v -run ^TestIntegration/TestDiffSingleLayer.*$ github.com/moby/buildkit/client -count=1 to run affected tests in buildkit with the tested runc in /usr/bin/runc.