Investigate (and implement) if it's possible to run the worker in non-privileged mode

[patrickdevivo]

For some users, running the worker container in privileged mode is a no-go. We should investigate if and how we can allow for non-privileged mode use with our container-sync approach (podman in a container)

[riyaz-ali]

It doesn't seem possible in the current iteration.

In order to run a container engine like Podman within a container, the first thing you need to understand is that you need a fair amount of privilege.

• Containers require multiple UIDs. Most container images need more than one UID to work. For example, you might have an image with most of the files owned by root, but some owned by the apache user (UID=60).
• Container engines mount file systems and use the system call clone to create user namespaces.

[source]

Alternative workarounds using CAP_SYS_ADMIN also doesn't seem to work. When running the worker with:

docker run --rm -it --cap-add=sys_admin --cap-add mknod --device=/dev/fuse --security-opt seccomp=unconfined --security-opt label=disable mergestat/worker /bin/sh

and then running podman like:

podman run --rm -it --network host alpine echo hi

still fails because /sys/fs/croup is mounted as read-only (and I couldn't figure out a way to fix it).

Even running as root without --privileged fails 🙄

For now, the only option seems to go with --privileged

[riyaz-ali]

Useful links

• https://github.com/containers/podman/blob/main/rootless.md
• https://github.com/containers/podman/blob/main/troubleshooting.md
• https://www.redhat.com/sysadmin/podman-inside-container

[amenowanna]

I was never able to get this to work in Docker without privileged mode. In K8s I was able to get it to work but it still required elevated permissions and not much different that running it in privileged mode. We will have to revisit this again at some point if the community in general is not supportive of this. We have had some users not be comfortable with this requirement in their production so we need to keep our eyes open for alternatives here.

[cvhariharan]

I have made some test changes that make it possible to run it without privileged mode. I have rewritten the podman controls to use docker api. #1160