Broken permissions in /var/lib/docker directory inside a workspace

[mwsv]

I'm using Sysbox to use Docker inside a workspace, workspace resource in Terraform looks like this:

resource "docker_container" "workspace" {
  count    = data.coder_workspace.me.start_count
  image    = docker_image.coder_image.image_id
  name     = "workspace-${lower(data.coder_workspace.me.name)}"
  hostname = data.coder_workspace.me.name

  privileged = true

  env     = ["CODER_AGENT_TOKEN=${coder_agent.dev.token}"]
  command = ["sh", "-c", coder_agent.dev.init_script]
  runtime = "sysbox-runc"

  host {
    host = "host.docker.internal"
    ip   = "host-gateway"
  }

  volumes {
    container_path = "/home/coder"
    volume_name    = docker_volume.home_volume.name
    read_only      = false
  }

  volumes {
    container_path = "/var/lib/docker"
    volume_name    = docker_volume.app_volume.name
    read_only      = false
  }

  mounts {
   target    = "/data"
   source    = "/coder/data/project-files"
   type      = "bind"
   read_only = false
  }

  labels {
    label = "coder.owner"
    value = data.coder_workspace.me.owner
  }
  labels {
    label = "coder.owner_id"
    value = data.coder_workspace.me.owner_id
  }
  labels {
    label = "coder.workspace_id"
    value = data.coder_workspace.me.id
  }
  labels {
    label = "coder.workspace_name"
    value = data.coder_workspace.me.name
  }
}

...it's used in on-premise instance of Coder. Unfortunately, I cannot run some of the workspaces from time to time, especially after Coder host machine is restarted. I have such an error message when I try to run dockerd inside a workspace:

...and I have noticed, that in such case permissions of /var/lib/docker directory inside a workspace are broken (nobody/nogroup user and group, I have got it only in such case):

...I would appreciate any suggestions/solutions for this.

[matifali]

When you use sysbox runtime, you are not supposed to mount the host docker docker into the workspace.
https://coder.com/docs/v2/latest/templates/docker-in-workspaces#sysbox-container-runtime

[mwsv]

Do you mean creating volume for /var/lib/docker directory? If yes, then how can I keep all docker data (databases, etc.) from the workspace interior when the workspace is restarted?

[matifali]

@mwsv Sorry, I misunderstood. You are doing that part correctly. Can you remove privileged = true as this is not sysbox-runc based Docker-in-Docker setup?
Also, ensure your user container user is a member of the docker group inside your Dockerfile.

[mwsv]

@matifali user coder inside a workspace belongs to docker group:

groups coder
coder : coder docker

...about privileged = true, it was an attempt to do something to fix this issue, with no effect of course. So it is not related with the issue - I had the same issue without this in template configuration file.

[ccll]

Same here, I ran the workspace container with sysbox runtime, and I created a volume to hold the workspace's /var/lib/docker to cache the docker images and docker build layers, to let them persistent across workspace rebuilds.

resource "docker_container" "workspace" {
  count = data.coder_workspace.me.start_count

  # Use image id instead of name, to recreate the container if the image changed.
  image = docker_image.main.image_id

  # Uses lower() to avoid Docker restriction on container names.
  name = "coder-${data.coder_workspace.me.owner}-${lower(data.coder_workspace.me.name)}"

  # Hostname makes the shell more user friendly: coder@my-workspace:~$
  hostname = data.coder_workspace.me.name

  # Use the docker gateway if the access URL is 127.0.0.1
  entrypoint = ["sh", "-c", replace(coder_agent.main.init_script, "/localhost|127\\.0\\.0\\.1/", "host.docker.internal")]

  # Use the Sysbox container runtime to enable using docker in workspace container.
  runtime = "sysbox-runc"

  env = ["CODER_AGENT_TOKEN=${coder_agent.main.token}"]
  host {
    host = "host.docker.internal"
    ip   = "host-gateway"
  }

  depends_on = [docker_image.main]

  // Mount vscode volume.
  volumes {
    container_path = "${local.home_folder}/.vscode-server"
    volume_name    = docker_volume.vscode_server_volume.name
    read_only      = false
  }

  // Mount workspace volume.
  volumes {
    container_path = "/workspace"
    volume_name    = docker_volume.workspace_volume.name
    read_only      = false
  }

  // Mount docker-root volume.
  volumes {
    container_path = "/var/lib/docker"
    volume_name    = docker_volume.docker_root.name
    read_only      = false
  }
}

The container user is root instead of coder.

At first it works great, but one day after I changed Dockerfile a bit (install opencv libraries in it) and rebuilt the workspace, the dockerd failed to start:

time="2024-05-09T02:40:15.029720561Z" level=info msg="containerd successfully booted in 0.046207s"
time="2024-05-09T02:40:16.045388913Z" level=info msg="stopping healthcheck following graceful shutdown" module=libcontainerd
time="2024-05-09T02:40:16.045469681Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plugins.moby
failed to start daemon: error setting up default driver: chown /var/lib/docker/volumes: operation not permitted

The owner of /var/lib/docker/volumes and some folders in /var/lib/docker/overlays mysteriously changed to nobody:nogroup, and the root user does not have access to those folders.

$ ll /var/lib/docker
total 132K
drwx--x--x   4 root   root    4.0K May  6 03:14 buildkit
drwx--x--x   3 root   root    4.0K May  6 03:14 containerd
drwx--x--- 191 root   root     20K May  8 07:50 containers
-rw-------   1 root   root      36 May  6 03:14 engine-id
drwx------   3 root   root    4.0K May  6 03:14 image
drwxr-x---   3 root   root    4.0K May  6 03:14 network
drwx--x--- 536 root   root     68K May  9 02:20 overlay2
drwx------   4 root   root    4.0K May  6 03:14 plugins
drwx------   2 root   root    4.0K May  9 02:55 runtimes
drwx------   2 root   root    4.0K May  6 03:14 swarm
drwx------   2 root   root    4.0K May  9 02:55 tmp
drwx-----x  15 nobody nogroup 4.0K May  8 07:46 volumes

$ ll /var/lib/docker/overlay2/
total 2.2M
drwx--x--- 4 root   root    4.0K May  7 04:54 0132246cc3628d64d2a325513fcf52f887cab6cd4b0dc969f07813b3ff6e6875
drwx--x--- 4 nobody nogroup 4.0K May  7 04:54 0132246cc3628d64d2a325513fcf52f887cab6cd4b0dc969f07813b3ff6e6875-init
drwx--x--- 4 nobody nogroup 4.0K May  6 03:24 0167db7b05b1c408988610fb13e46826c0129a032cd9920299f63a0936ada09a
drwx--x--- 4 nobody nogroup 4.0K May  6 03:24 0167db7b05b1c408988610fb13e46826c0129a032cd9920299f63a0936ada09a-init
drwx--x--- 4 nobody nogroup 4.0K May  7 05:52 01dc2778d9e415e1c7b26f1688d82a69d6ccf0772736026c36a2cfe438c92297
drwx--x--- 4 nobody nogroup 4.0K May  7 05:52 01dc2778d9e415e1c7b26f1688d82a69d6ccf0772736026c36a2cfe438c92297-init
drwx--x--- 4 nobody nogroup 4.0K May  7 06:46 032dab272b2c53e1bd78fdf8ca411d8d2f43ba8c484803712595761af2b26b1e
drwx--x--- 4 root   root    4.0K May  7 06:46 032dab272b2c53e1bd78fdf8ca411d8d2f43ba8c484803712595761af2b26b1e-init
drwx--x--- 4 nobody nogroup 4.0K May  8 08:01 03586348247eb14a7597f983f9c2e11e776b76078ad0bc2f60d041e838160c55
drwx--x--- 4 nobody nogroup 4.0K May  7 03:28 0418a1533af210aaf79ef7d4372a0dc10afe1f8a07e4e5d3125253b4c3a9c59c
drwx--x--- 4 nobody nogroup 4.0K May  7 03:28 0418a1533af210aaf79ef7d4372a0dc10afe1f8a07e4e5d3125253b4c3a9c59c-init
drwx--x--- 4 nobody nogroup 4.0K May  6 03:17 048d2dbf6e604293b9011a14ed82736ba7195059f6f94a198569ab66206a25c7
drwx--x--- 4 nobody nogroup 4.0K May  7 05:49 05647c4e1549a660ca4bb5d57e5e2ab717fd587cd2bdc9def0922227804c7a6f
drwx--x--- 4 nobody nogroup 4.0K May  7 05:49 05647c4e1549a660ca4bb5d57e5e2ab717fd587cd2bdc9def0922227804c7a6f-init
drwx--x--- 4 nobody nogroup 4.0K May  8 07:02 0567981f6d9a715b58e430ea6e0a8b3bb6c515d77d043076d180ae5e230e5a66
drwx--x--- 4 nobody nogroup 4.0K May  8 07:02 0567981f6d9a715b58e430ea6e0a8b3bb6c515d77d043076d180ae5e230e5a66-init
drwx--x--- 4 nobody nogroup 4.0K May  8 07:58 0666449aa0a5f72669757be93cb7b4283fb8b3a2a45006fada28f61463f0a6f0
drwx--x--- 4 nobody nogroup 4.0K May  7 05:38 067fb81937c2b7fc1af3107e28b7fa8e1357ba4a6134ce250e446230adbfed86
drwx--x--- 4 root   root    4.0K May  6 04:07 07934d5cbcef6e13cff6e5145117366da0e4f5cc1ace7a3ff67ef45ff01e64a3
drwx--x--- 4 root   root    4.0K May  6 04:07 07934d5cbcef6e13cff6e5145117366da0e4f5cc1ace7a3ff67ef45ff01e64a3-init
drwx--x--- 4 nobody nogroup 4.0K May  7 06:44 07c958290289b59f5f04eba9812db2b9798e8d6ab73cd8c5f77c03e82d1c3f8f
drwx--x--- 4 nobody nogroup 4.0K May  7 06:44 07c958290289b59f5f04eba9812db2b9798e8d6ab73cd8c5f77c03e82d1c3f8f-init
...

$ ll /var/lib/docker/volumes/
ls: cannot open directory '/var/lib/docker/volumes/': Permission denied

Entirely delete and recreate a new workspace works for some time, then the same problems would appear again after an occasional restart.

I'm wondering if the problem rooted in Coder, or Sysbox, or the Docker daemon on the host?

[ccll]

Just found an interesting doc from Sysbox.

• https://github.com/nestybox/sysbox/blob/master/docs/user-guide/storage.md#mounting-storage-with-docker--sysbox

Check the uid:gid from host:

root@homelab-1 /mnt/nvme/2/docker/volumes/coder-b5458a57-9004-4471-ba1e-ca3ff1deb404-docker-root/_data $  ll
total 132K
drwx--x--x   4 1345184 1345184 4.0K May  6 11:14 buildkit/
drwx--x--x   3 1345184 1345184 4.0K May  6 11:14 containerd/
drwx--x--- 191 1345184 1345184  20K May  8 15:50 containers/
-rw-------   1 1345184 1345184   36 May  6 11:14 engine-id
drwx------   3 1345184 1345184 4.0K May  6 11:14 image/
drwxr-x---   3 1345184 1345184 4.0K May  6 11:14 network/
drwx--x--- 536 1345184 1345184  68K May  9 10:20 overlay2/
drwx------   4 1345184 1345184 4.0K May  6 11:14 plugins/
drwx------   2 1345184 1345184 4.0K May  9 11:04 runtimes/
drwx------   2 1345184 1345184 4.0K May  6 11:14 swarm/
drwx------   2 1345184 1345184 4.0K May  9 11:04 tmp/
drwx-----x  15 2690368 2690368 4.0K May  8 15:46 volumes/

root@homelab-1 /mnt/nvme/2/docker/volumes/coder-b5458a57-9004-4471-ba1e-ca3ff1deb404-docker-root/_data $  cat /etc/subuid
...
htpc:165536:65536
clash:231072:65536
containers:296608:1048576
sysbox:1345184:65536
collector:1410720:65536
steam:1476256:65536

Sysbox uid starts from 1345184, which is the root user (id=0) in the container, looks correct.
But the uid of volumes folder 2690368 is way out of the range of [1345184, 1345184+65536].

After stopping all the workspace containers and inspect the volumes folder from the host:

$ ll *-docker-root/*
...
coder-a247d520-fade-47ab-a74e-b4e358b93890-inner-docker-root/_data:
total 48K
drwx--x--x  5 root root 4.0K Apr 29 09:56 buildkit
drwx--x--x  3 root root 4.0K Apr 29 09:53 containerd
drwx--x---  7 root root 4.0K Apr 29 09:59 containers
-rw-------  1 root root   36 Apr 29 09:53 engine-id
drwx------  3 root root 4.0K Apr 29 09:53 image
drwxr-x---  3 root root 4.0K Apr 29 09:53 network
drwx--x--- 48 root root 4.0K May  9 10:39 overlay2
drwx------  4 root root 4.0K Apr 29 09:53 plugins
drwx------  2 root root 4.0K May  9 10:39 runtimes
drwx------  2 root root 4.0K Apr 29 09:53 swarm
drwx------  2 root root 4.0K May  9 10:39 tmp
drwx-----x  2 root root 4.0K Apr 29 09:53 volumes

coder-b5458a57-9004-4471-ba1e-ca3ff1deb404-docker-root/_data:
total 132K
drwx--x--x   4 4293622112 4293622112 4.0K May  6 11:14 buildkit
drwx--x--x   3 4293622112 4293622112 4.0K May  6 11:14 containerd
drwx--x--- 191 4293622112 4293622112  20K May  8 15:50 containers
-rw-------   1 4293622112 4293622112   36 May  6 11:14 engine-id
drwx------   3 4293622112 4293622112 4.0K May  6 11:14 image
drwxr-x---   3 4293622112 4293622112 4.0K May  6 11:14 network
drwx--x--- 536 4293622112 4293622112  68K May  9 10:20 overlay2
drwx------   4 4293622112 4293622112 4.0K May  6 11:14 plugins
drwx------   2 4293622112 4293622112 4.0K May  9 11:41 runtimes
drwx------   2 4293622112 4293622112 4.0K May  6 11:14 swarm
drwx------   2 4293622112 4293622112 4.0K May  9 11:41 tmp
drwx-----x  15 root       root       4.0K May  8 15:46 volumes
...

We can see that the uid:gid in the problematic workspace volume (b5458a57-9004-4471-ba1e-ca3ff1deb404) is very off compared to other normal workspaces.

I'm not an expert in Docker and container and namespaces all these stuff, so I don't know why this is happening, I hope this would provide clues for someone to find out.

For now I could fix the problem temporarily by chown -R root:root /path/to/coder/volume from THE HOST while the Coder workspace container is STOPPED, as all the other workspace volume folders/files seem to have root:root ownership on the host.