DOCKER_DEFAULT_PLATFORM breaks kind if it does not match the dockerd host platform

[subnetmarco]

When running kind create cluster it fails with the following log message:

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✗ Preparing nodes 📦
ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

Environment:

• kind version:

$ kind version
kind v0.12.0 go1.17.8 darwin/arm64

• Kubernetes version:

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.2", GitCommit:"8b5a19147530eaac9476b0ab82980b4088bbc1b2", GitTreeState:"clean", BuildDate:"2021-09-15T21:31:32Z", GoVersion:"go1.16.9", Compiler:"gc", Platform:"darwin/arm64"}

• Docker version: (use docker info):

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.4.1)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 10
 Server Version: 20.10.14
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3df54a852345ae127d1fa3092b95168e4a88e2f8
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 5
 Total Memory: 7.765GiB
 Name: docker-desktop
 ID: LZ6O:NZMG:OHAC:45JC:ED7V:RX4B:O3X5:ENMG:CQ76:QAKS:S2UV:HWHJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

• OS:

macOS Monterey (v12.0) on Apple M1 Pro.

[aojea]

a similar one reported by fedora seems that was caused by the kernel version
#2689

[lukevs]

I was experiencing the same issue
Fixed for me by creating an architecture-specific image

Dockerfile

FROM --platform=arm64 kindest/node:v1.23.4
RUN arch

Built as

docker build -t tempkind .

Then

kind create cluster --image tempkind

[aojea]

hmm, we are publishing arm images https://hub.docker.com/layers/node/kindest/node/v1.23.4/images/sha256-0415a7bb1275c23c9315d97c5bfd7aecfdf8a8ea0562911f653711d947f7bec0?context=explore 🤔

[stmcginnis]

Originally reported issue doesn't appear to be related to availability of an image for arm64:

ERROR: failed to create cluster: could not find a log line that matches "Reached target .*Multi-User System.*|detected cgroup v1"

I have also created kind clusters on an M1 Mac successfully. I don't think this specific version, but normal creation works fine. Marco's issue appears to be something triggering a failure during the use of that image.

[subnetmarco]

I confirm that following @lukevs instructions actually works. I created a new Docker build as he suggested, and the Kubernetes cluster started properly:

$ kind create cluster --image tempkind
Creating cluster "kind" ...
 ✓ Ensuring node image (tempkind) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/

[BenTheElder]

That doesn't make any sense, why isn't docker selecting the arm image from the multi-arch manifest? 🤔

What version of the docker desktop app is this? (Not dockerd / docker CLI, check the GUI / tray icon about info)

[subnetmarco]

Docker Desktop Version: 4.7.0 (77141)
Engine: 20.10.14

[BenTheElder]

Is this behavior reproducible with other multi arch images?

This feels like a regression in the docker app, we know that M1 worked before and we didn't change much related to that on our end. We should double check the full default image reference in v0.12.0 though.

[BenTheElder]

The only other explanation I have is v0.12 being bugged with the digest pointing to the amd64 image specifically. Will check for that.
The image reference is truncated when printing the pull info unless more verbose logging is used IIRC.

[BenTheElder]

For v0.12.0 the default image is ef21f80#diff-643b1e9d9e446aa30da4407354de0098f24c947ac985213a06f73188c3e8e3fcR21
=> "kindest/node:v1.23.4@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9"

It's still crane digest kindest/node:v1.23.4 => sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0df

And if you inspect that image you get:

{
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
   "manifests": [
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 743,
         "digest": "sha256:1742ff7f0b79a8aaae347b9c2ffaf9738910e721d649301791c812c162092753",
         "platform": {
            "architecture": "amd64",
            "os": "linux"
         }
      },
      {
         "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
         "size": 742,
         "digest": "sha256:0415a7bb1275c23c9315d97c5bfd7aecfdf8a8ea0562911f653711d947f7bec0",
         "platform": {
            "architecture": "arm64",
            "os": "linux"
         }
      }
   ]
}

So docker should be auto-selecting an arm64 image on M1 ....

[BenTheElder]

I don't currently have docker on M1, will need help narrowing down the source of this behavior #2718 (comment)

[subnetmarco]

Just as an FYI, Docker Desktop released a new version 4.7.1 (77678) but this problem still happens.

[BenTheElder]

#2718 (comment) does docker run some-multi-arch-image also result in running the amd64 image instead of arm64?

@aojea and I cannot test this ourselves at the moment. But this workaround suggests a docker bug.

[stmcginnis]

Just tested on a MacBook Air M1 - macOS 12.3.1

$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.23.4) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

$ kind version
kind v0.12.0 go1.17.8 darwin/arm64

$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.1)
  compose: Docker Compose (Docker Inc., v2.3.3)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 292
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2 io.containerd.runtime.v1.linux
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.104-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 6
 Total Memory: 3.826GiB
 Name: docker-desktop
 ID: OE44:7GZ6:JZKY:H6LU:LIFW:HL3U:2A37:4SG4:7QDJ:ZEWN:UWMY:NCL2
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5000
  127.0.0.0/8
 Live Restore Enabled: false

Looks like docker info returns the same kernel version as shown the original failure description.

[stmcginnis]

$ docker images | grep kindest
kindest/node                                                                  <none>             f9d67d6e8342   6 weeks ago     1.21GB

$ docker inspect f9d67d6e8342
[
    {
        "Id": "sha256:f9d67d6e83427c29b4c9d413c2d79c5d5a5f3da57c39e59a0c892c183812624f",
        "RepoTags": [],
        "RepoDigests": [
            "kindest/node@sha256:0e34f0d0fd448aa2f2819cfd74e99fe5793a6e4938b328f657c8e3f81ee0dfb9"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "2022-03-06T21:44:36.257726427Z",
        "Container": "8ab6159dddf292aa03d0763d3ba61c4735fefcf6ed906bffaadc403bab93d4eb",
        "ContainerConfig": {
            "Hostname": "8ab6159dddf2",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=docker"
            ],
            "Cmd": [
                "infinity"
            ],
            "Image": "docker.io/kindest/base:v20220305-b67a383f",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "sleep"
            ],
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGRTMIN+3"
        },
        "DockerVersion": "20.10.11",
        "Author": "",
        "Config": {
            "Hostname": "8ab6159dddf2",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "container=docker"
            ],
            "Cmd": null,
            "Image": "docker.io/kindest/base:v20220305-b67a383f",
            "Volumes": null,
            "WorkingDir": "/",
            "Entrypoint": [
                "/usr/local/bin/entrypoint",
                "/sbin/init"
            ],
            "OnBuild": null,
            "Labels": {},
            "StopSignal": "SIGRTMIN+3"
        },
        "Architecture": "arm64",
        "Os": "linux",
        "Size": 1214940253,
        "VirtualSize": 1214940253,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/a736ff8cd71c7d07cdced41e53aad3d2b44dc98265b24e51baac8cff0f6eae2a/diff",
                "MergedDir": "/var/lib/docker/overlay2/4fe148cc1e4c4b1539bbdc023b3033667b3f501d9ec3caea6185f81585d8ca45/merged",
                "UpperDir": "/var/lib/docker/overlay2/4fe148cc1e4c4b1539bbdc023b3033667b3f501d9ec3caea6185f81585d8ca45/diff",
                "WorkDir": "/var/lib/docker/overlay2/4fe148cc1e4c4b1539bbdc023b3033667b3f501d9ec3caea6185f81585d8ca45/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:d384c88eae37978ca8a95aff7b13c6987b0bdedec50fca7890c484b0918ccb73",
                "sha256:db19631aac508ec95983ced710b2ca045ec593603644e6b31f833bfaae47092b"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        }
    }
]

[jeff-mccoy]

Same. Just ran on 12.3 with Kind 0.12.0 and couldn't reproduce.

[svmaris]

I had the same problem, but I then remembered I had added DOCKER_DEFAULT_PLATFORM=linux/amd64 to my ~/.zshrc. After unsetting the env-var it would still throw the same error. Only after removing the images completely (docker image rm ...) Docker would pull the correct image and the problem was gone.

[subnetmarco]

@svmaris thank you! This was exactly my problem. I removed the environment variable, deleted the Docker images, and now it works like a charm.

[BenTheElder]

I had the same problem, but I then remembered I had added DOCKER_DEFAULT_PLATFORM=linux/amd64 to my ~/.zshrc. After unsetting the env-var it would still throw the same error. Only after removing the images completely (docker image rm ...) Docker would pull the correct image and the problem was gone.

Thank you! That explains everything, including why the workaround of building an arm64 only image would work #2718 (comment)

We should probably consider some follow up here ...

Is this env only read by the docker CLI? It seems so given your docker info shows aarch64 on the server. If that's the case, kind should maybe unset this when invoking docker

If not .. we should consider forcing --platform to be the server platform all over the place ... which would be pretty messy.

At minimum we should probably add a known issues entry if we don't plan to work around this.

[BenTheElder]

Is this env only read by the docker CLI? It seems so given your docker info shows aarch64 on the server. If that's the case, kind should maybe unset this when invoking docker

Answering myself: yes https://docs.docker.com/engine/reference/commandline/cli/

DOCKER_DEFAULT_PLATFORM | Default platform for commands that take the --platform flag.

We should consider if it makes sense for kind to block this env from reaching docker calls invoked by kind, or else to check and warn if it doesn't seem to match the server info when doing certain operations (kind create cluster is probably the only one in particular).

I think it might make sense to have kind ignore broadly, like unsetting it in the kind binary or filtering it in execs to docker. Using exec(docker ...) is a current implementation detail, if we intended to support this #2735 we'd have some kind specific option probably (KIND_DEFAULT_PLATFORM perhaps), and we don't, as it needs to match the host platform to work.

[ungps]

Had the same issue above; but @lukevs solution didn't work for me, I was getting some other error:

Command Output: Unable to find image 'tmpkind' locally
docker: Error response from daemon: pull access denied for tmpkind, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

Fixed it by setting DOCKER_BUILDKIT=1 as env variable. You also have to build the image with this env variable.

DOCKER_BUILDKIT=1 kind create cluster --image tmpkind
Creating cluster "kind" ...
 ✓ Ensuring node image (scgckind) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a question, bug, or feature request? Let us know! https://kind.sigs.k8s.io/#community 🙂

[cavokz]

Is this env only read by the docker CLI? It seems so given your docker info shows aarch64 on the server. If that's the case, kind should maybe unset this when invoking docker

Answering myself: yes https://docs.docker.com/engine/reference/commandline/cli/

DOCKER_DEFAULT_PLATFORM | Default platform for commands that take the --platform flag.

We should consider if it makes sense for kind to block this env from reaching docker calls invoked by kind, or else to check and warn if it doesn't seem to match the server info when doing certain operations (kind create cluster is probably the only one in particular).

I think it might make sense to have kind ignore broadly, like unsetting it in the kind binary or filtering it in execs to docker. Using exec(docker ...) is a current implementation detail, if we intended to support this #2735 we'd have some kind specific option probably (KIND_DEFAULT_PLATFORM perhaps), and we don't, as it needs to match the host platform to work.

Don't know if there is any use case for setting DOCKER_DEFAULT_PLATFORM but preventing it to reach Docker would introduce some non-linear behavior in Kind. I think the user should be allowed to make mistakes like this.

I'm therefore in favor of just putting a warning in case of mismatching values.

[BenTheElder]

@ungps for now you should probably unset DOCKER_DEFAULT_PLATFORM when calling kind e.g. like DOCKER_DEFAULT_PLATFORM= kind create cluster, then it will work with the multi-arch images isntead.

---

Don't know if there is any use case for setting DOCKER_DEFAULT_PLATFORM but preventing it to reach Docker would introduce some non-linear behavior in Kind. I think the user should be allowed to make mistakes like this.

This is a fair point, though kind being implemented by way of exec(docker, ...) is a current leaky implementation detail FWIW, the client library wouldn't respect all these env AFAIK.

We could similarly instead accomplish this by setting --platform=$dockerd_server_platform when we call docker, but it's equivalent and simpler to just not pass the env through.

If we just do a warning, the user will have to actively alter their host configuration or at least override it when invoking kind, which is more friction.

A compromise might be detecting that the env is set and warning that kind ignores it and include a reference to this issue or some documentation.

It seems to be an unintuitive failure mode that took a month to pin down for the first user with more trickling in, so we should probably move towards one of these soon. cc @aojea
/priority important-soon

[cavokz]

This is a fair point, though kind being implemented by way of exec(docker, ...) is a current leaky implementation detail FWIW, the client library wouldn't respect all these env AFAIK.

I overlooked this, then maybe it's better to completely sanitize the environment and let only the explicitly configured variables.

[aojea]

what is the reasons of setting DOCKER_DEFAULT_PLATFORM ?

is this because of this https://stackoverflow.com/questions/65612411/forcing-docker-to-use-linux-amd64-platform-by-default-on-macos?

that seems a temporary problem , if affirmative I think warning and link is the best way to proceed?

[BenTheElder]

what is the reasons of setting DOCKER_DEFAULT_PLATFORM ?
is this because of this https://stackoverflow.com/questions/65612411/forcing-docker-to-use-linux-amd64-platform-by-default-on-macos?

I think this is the primary reason

that seems a temporary problem , if affirmative I think warning and link is the best way to proceed?

I don't see any reason it would be temporary, as long as users are still developing amd64-only projects from macOS, where the host platform will be heading to arm64 + emulation for the forseeable future.

---

I overlooked this, then maybe it's better to completely sanitize the environment and let only the explicitly configured variables.

Yeah, I think as a general rule we probably should sanitize env going to exec, this particular issue just seems to have highlighted it.

If, in the future, this setting were actually supportable, we could consider intentionally passing through the env.

[diranged]

Just chiming in here... perhaps I should open a new issue, but it would be really powerful for us to be able to launch a KIND cluster on our M1 Mac's with both AMD64 and ARM64 Nodes. We are struggling to reproduce certain environments locally (like istio, which doesn't yet have a mullti-arch proxyv2 image), and being able to run a full multi-architecture local kind cluster would be an incredible value. Is that something that is remotely feasible in the future?

[BenTheElder]

Just chiming in here... perhaps I should open a new issue, but it would be really powerful for us to be able to launch a KIND cluster on our M1 Mac's with both AMD64 and ARM64 Nodes. We are struggling to reproduce certain environments locally (like istio, which doesn't yet have a mullti-arch proxyv2 image), and being able to run a full multi-architecture local kind cluster would be an incredible value. Is that something that is remotely feasible in the future?

Unfortunately not. See #2738 (comment)

The way docker run does multi-arch cannot run complex workloads like Kubernetes, only simple compute with limited syscalls.
To run multi arch Kubernetes you need a host / kernel (could be a VM) actually running on each arch.

I would posit that you should be able to test with two separate clusters one of each arch though.

Docker desktop already employs a VM to get linux on Mac. In theory a second VM on amd64 would let you run an amd64 cluster. I haven't seen anyone make this work yet though. I think most tools don't support x86 VMs on M1 because it would be slow without x86 hardware virtualization support.

We do not have any intent of supporting multi-machine (you should use an actual non-local cluster installer for this) so it wouldn't be a single cluster using two VMs to host the respective architectures, and the qemu bimft_misc hack cannot run Kubernetes.

[rajjaiswalsaumya]

The simplest approach is to export DOCKER_DEFAULT_PLATFORM=linux/arm64 for Mac M1 or just empty that field as DOCKER_DEFAULT_PLATFORM=, and then run kind create cluster --config create-kind-cluster.yaml This fixed my issue.

I used the create-kind.yaml file as

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
- role: worker
  image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
- role: worker
  image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea
- role: worker
  image: kindest/node-arm64:v1.26.3@sha256:3dc28d458c92e3252c78752ade134d7231ce8e3565f2de4e437a77b171d046ea