opentelemetry-resource-detector-container does not detect the container id on 0.3.4/0.3.7, but works on 0.2.2

[harold-kfuse]

I'm running on AWS EC2 instance (Amazon linux image). This is the docker version:

docker --version
Docker version 24.0.5, build ced0996

I deployed otel-demo. The front-end is has dependency with 0.3.4. But the ContainerDetector is not able to detect the container id.

I upgraded the version to 0.3.7 and it still does not work. I downgraded to 0.2.2, it is able to detect the container id.

These are the content of /proc/self/cgroup and /proc/self/mountinfo

[ec2-user@ip-172-31-18-222 opentelemetry-demo]$ docker exec 2a2be61a2ae4 cat /proc/self/cgroup
0::/

[ec2-user@ip-172-31-18-222 opentelemetry-demo]$ docker exec 2a2be61a2ae4 cat /proc/self/mountinfo
1901 1716 0:220 / / rw,relatime master:602 - overlay overlay rw,seclabel,lowerdir=/var/lib/docker/overlay2/l/ZPJFPFVUVQTN2BOMZ6HP2OBVAP:/var/lib/docker/overlay2/l/VKAARZVBXIVTH2IDMMA6XZYF6L:/var/lib/docker/overlay2/l/E7TKPSHIJTXXLKHC7QNOW5XZTH:/var/lib/docker/overlay2/l/LD7K7265HP3HWGJG3LKQ36F6WK:/var/lib/docker/overlay2/l/2IQCGLNNNRFFMBNLUPNPLA3QTV:/var/lib/docker/overlay2/l/PD5L245VNZ4MQVEVJNQLBEPMT6:/var/lib/docker/overlay2/l/HAAEZHJS3TXADWQI5Z7YYVJDFI:/var/lib/docker/overlay2/l/TEN4625OQ2Z62WA7GJTJBVSTF4:/var/lib/docker/overlay2/l/LV7F4HHQWNALHB434L2RX62AK2:/var/lib/docker/overlay2/l/GRWWTLVD5CZVE3IH5OLB4HEBHH:/var/lib/docker/overlay2/l/BNDNOKLAOJPATBZI7HCEM423CW:/var/lib/docker/overlay2/l/SS6FB6CPCXAAQLNJ3JVCV2GQZJ:/var/lib/docker/overlay2/l/7JCI4G44JFF2YAKKZ2Q4LNZFDJ:/var/lib/docker/overlay2/l/GHHJD6E6IDLZ4SXRSQKKGELPJH:/var/lib/docker/overlay2/l/HWDHCCEOL4VL27MBJ62P7FNLYQ:/var/lib/docker/overlay2/l/6VQO6OLMJE4ZZCL2Z7N47VZDKC,upperdir=/var/lib/docker/overlay2/81f8e487f7a720f91e47b4ecffe129e38dc19d1840209a96e9d1446bcfb2b06a/diff,workdir=/var/lib/docker/overlay2/81f8e487f7a720f91e47b4ecffe129e38dc19d1840209a96e9d1446bcfb2b06a/work
1902 1901 0:223 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
1903 1901 0:224 / /dev rw,nosuid - tmpfs tmpfs rw,seclabel,size=65536k,mode=755
1904 1903 0:225 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,seclabel,gid=5,mode=620,ptmxmode=666
1905 1901 0:226 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro,seclabel
1906 1905 0:23 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw,seclabel
1907 1903 0:222 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw,seclabel
1908 1903 0:227 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,seclabel,size=65536k
1909 1901 202:1 /var/lib/docker/containers/2a2be61a2ae405bb2e33bb8835551d98a68e90274e0fdf416005a54ba2324bb1/resolv.conf /etc/resolv.conf rw,noatime - xfs /dev/xvda1 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota
1910 1901 202:1 /var/lib/docker/containers/2a2be61a2ae405bb2e33bb8835551d98a68e90274e0fdf416005a54ba2324bb1/hostname /etc/hostname rw,noatime - xfs /dev/xvda1 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota
1911 1901 202:1 /var/lib/docker/containers/2a2be61a2ae405bb2e33bb8835551d98a68e90274e0fdf416005a54ba2324bb1/hosts /etc/hosts rw,noatime - xfs /dev/xvda1 rw,seclabel,attr2,inode64,logbufs=8,logbsize=32k,sunit=1024,swidth=1024,noquota
1717 1902 0:223 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
1753 1902 0:223 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
1754 1902 0:223 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
1755 1902 0:223 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
1756 1902 0:223 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
1757 1902 0:228 / /proc/acpi ro,relatime - tmpfs tmpfs ro,seclabel
1758 1902 0:224 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,seclabel,size=65536k,mode=755
1759 1902 0:224 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,seclabel,size=65536k,mode=755
1780 1902 0:224 /null /proc/latency_stats rw,nosuid - tmpfs tmpfs rw,seclabel,size=65536k,mode=755
1781 1902 0:224 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,seclabel,size=65536k,mode=755
1782 1902 0:229 / /proc/scsi ro,relatime - tmpfs tmpfs ro,seclabel
1795 1905 0:230 / /sys/firmware ro,relatime - tmpfs tmpfs ro,seclabel

[harold-kfuse]

cc @abhee11

[abhee11]

I will take this issue up .

[trentm]

Looking a little bit at the history of @opentelemetry/resource-detector-container: The only logic changes to that package between v0.22.0 and v0.3.4 is:

• 5675c49 - fix: Update defect of of wrong resource attribute of "container.id" (fix: Update defect of of wrong resource attribute of "container.id" #1682)

However, that change only touched the code that is parsing /proc/self/cgroup for cgroupsv1 support. The example here has the mostly-empty /proc/self/cgroup that one gets on cgroupsv2.

Is it possible that changes in #1682 made it so _getContainerIdV1() returns a value for container.id with that mostly-empty content:

[ec2-user@ip-172-31-18-222 opentelemetry-demo]$ docker exec 2a2be61a2ae4 cat /proc/self/cgroup
0::/

? If so, that could result in this kind of breakage.

[tobernguyen]

I'm on the latest version (0.3.8) and it doesn't detect container id

/app $ cat /proc/self/cgroup
0::/

/app $ cat /proc/self/mountinfo
4659 3172 0:976 / / rw,relatime master:446 - overlay overlay rw,lowerdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/215/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/125/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/124/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/123/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/122/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/121/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/120/fs:/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/15/fs,upperdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/218/fs,workdir=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/218/work,uuid=on
4660 4659 0:987 / /proc rw,nosuid,nodev,noexec,relatime - proc proc rw
4661 4659 0:988 / /dev rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
4662 4661 0:1181 / /dev/pts rw,nosuid,noexec,relatime - devpts devpts rw,gid=5,mode=620,ptmxmode=666
4663 4661 0:970 / /dev/mqueue rw,nosuid,nodev,noexec,relatime - mqueue mqueue rw
4664 4659 0:974 / /sys ro,nosuid,nodev,noexec,relatime - sysfs sysfs ro
4665 4664 0:30 / /sys/fs/cgroup ro,nosuid,nodev,noexec,relatime - cgroup2 cgroup rw
4666 4659 254:1 /docker/volumes/b6d29c4b8a87b7ca942dfc4e4e048ea658a239049741a61afe5a1c016bf0601b/_data/pods/f85e07e5-cf55-4320-8c37-795f3b991178/etc-hosts /etc/hosts rw,relatime - ext4 /dev/vda1 rw,discard
4667 4661 254:1 /docker/volumes/b6d29c4b8a87b7ca942dfc4e4e048ea658a239049741a61afe5a1c016bf0601b/_data/pods/f85e07e5-cf55-4320-8c37-795f3b991178/containers/demo-otel/cf6fa2b8 /dev/termination-log rw,relatime - ext4 /dev/vda1 rw,discard
4668 4659 254:1 /docker/volumes/4d50ad9be5c7aa90d74306fd9cc315242fd3ad0fb9cfaafeac65891f275c618b/_data/agent/containerd/io.containerd.grpc.v1.cri/sandboxes/26ab5944047bd9c4703e80aa03ba989c6b5e3dc435fd664cd219034b07a32f7e/hostname /etc/hostname rw,relatime - ext4 /dev/vda1 rw,discard
4669 4659 254:1 /docker/volumes/4d50ad9be5c7aa90d74306fd9cc315242fd3ad0fb9cfaafeac65891f275c618b/_data/agent/containerd/io.containerd.grpc.v1.cri/sandboxes/26ab5944047bd9c4703e80aa03ba989c6b5e3dc435fd664cd219034b07a32f7e/resolv.conf /etc/resolv.conf rw,relatime - ext4 /dev/vda1 rw,discard
4670 4661 0:809 / /dev/shm rw,nosuid,nodev,noexec,relatime - tmpfs shm rw,size=65536k
4671 4659 0:764 / /run/secrets/kubernetes.io/serviceaccount ro,relatime - tmpfs tmpfs rw,size=24576808k
3173 4660 0:987 /bus /proc/bus ro,nosuid,nodev,noexec,relatime - proc proc rw
3174 4660 0:987 /fs /proc/fs ro,nosuid,nodev,noexec,relatime - proc proc rw
3175 4660 0:987 /irq /proc/irq ro,nosuid,nodev,noexec,relatime - proc proc rw
3176 4660 0:987 /sys /proc/sys ro,nosuid,nodev,noexec,relatime - proc proc rw
3177 4660 0:987 /sysrq-trigger /proc/sysrq-trigger ro,nosuid,nodev,noexec,relatime - proc proc rw
3178 4660 0:988 /null /proc/kcore rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
3179 4660 0:988 /null /proc/keys rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
3180 4660 0:988 /null /proc/timer_list rw,nosuid - tmpfs tmpfs rw,size=65536k,mode=755
3181 4664 0:1182 / /sys/firmware ro,relatime - tmpfs tmpfs ro

I can confirm that 0.2.2 can detect a container ID but I'm not sure if it did so correctly. The detected container.id will be 4d50ad9be5c7aa90d74306fd9cc315242fd3ad0fb9cfaafeac65891f275c618b which didn't match what was in the .status.containerStatuses[0].containerID