ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry" #44570

thaJeztah · 2022-12-02T09:06:00Z

Description

Since late yesterday evening, BuildKit CI is broken; as far as I can see, it's breaking in a step that does not use the code in this repository (but please double-check if I'm right), so either

The base image used
The Dockerfile frontend (it looks to be using dockerfile-upstream:master)
Perhaps some obscure issue, similar to including git attributes in "vendor" makes archive checksum change containerd/containerd#6387, where the Git "short commit" is used in the source code?
Corrupted remote build-cache? (does it use that?)
something changes on GitHub action hosts ?
GitHub actions builds of docker / containerd / buildx ?

Failure looks like this: (https://github.com/moby/moby/actions/runs/3600129177/jobs/6064524682)

> [containerd-alt-15 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.5.11"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out:
#0 4.935 + bin/containerd
#47 268.1 # github.com/containerd/containerd/vendor/github.com/miekg/pkcs11
#47 268.1 cgo: malformed DWARF TagVariable entry
#47 279.5 make: *** [Makefile:213: bin/containerd] Error 2

Which is similar to reports in go1.17 some time ago;

looks reported in cgo: malformed DWARF TagVariable entry golang/go#53000
and a fix in https://go-review.googlesource.com/c/go/+/406816 ?

Reproduce

See #44567 and other PRs

Expected behavior

No response

docker version

sudo rm /etc/docker/daemon.json
  sudo service docker restart
  docker version
  docker info
  shell: /usr/bin/bash -e {0}
  env:
    BUNDLES_OUTPUT: ./bundles
    BUILDKIT_REF: 4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
Client:
 Version:           20.10.21+azure-1
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f82a10204ec5708d5fbba130ad76cfee49
 Built:             Tue Oct 25 17:53:02 UTC 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

docker info

Server:
 Engine:
  Version:          20.10.21+azure-1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208812eb5e792fa99736c9167d1e10f4ab49
  Built:            Tue Oct 25 11:44:15 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.14+azure-1
  GitCommit:        b84d0b151c2395a5917996d602b192ce1e0fa461
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:        
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., 0.9.1+azure-2)
  compose: Docker Compose (Docker Inc., 2.12.2+azure-1)

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 21
 Server Version: 20.10.21+azure-1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b84d0b151c2395a5917996d602b192ce1e0fa461
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.15.0-1023-azure
 Operating System: Ubuntu 20.04.5 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 6.781GiB
 Name: fv-az77-14
 ID: 4IWW:VLI2:7YTM:AFEZ:4NJ5:VCDD:VCJU:3HXK:DSAL:C6S2:OKSN:N3A5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: githubactions
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

The step where things break:

shell: /usr/bin/bash -e {0}
  env:
    BUNDLES_OUTPUT: ./bundles
    BUILDKIT_REF: 4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
    CONTEXT: .
    TEST_DOCKERD: 1
    TEST_DOCKERD_BINARY: ./build/moby/binary-daemon/dockerd
    TESTPKGS: ./client
    TESTFLAGS: -v --parallel=1 --timeout=30m --run=//worker=dockerd$
+ docker buildx build --target integration-tests --output type=docker,name=buildkit-tests . --progress=plain

Failure with more detail:

 > [containerd-alt-15 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.5.11"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out:
#0 5.824 + bin/containerd
#49 311.7 # github.com/containerd/containerd/vendor/github.com/miekg/pkcs11
#49 311.7 cgo: malformed DWARF TagVariable entry
#49 324.2 make: *** [Makefile:213: bin/containerd] Error 2
------
Dockerfile:143
--------------------
 142 |     ARG GO111MODULE=off
 143 | >>> RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache \
 144 | >>>   git fetch origin \
 145 | >>>   && git checkout -q "$CONTAINERD_ALT_VERSION_15" \
 146 | >>>   && make bin/containerd \
 147 | >>>   && make bin/containerd-shim-runc-v2 \
 148 | >>>   && mv bin /out
 149 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c git fetch origin   && git checkout -q \"$CONTAINERD_ALT_VERSION_15\"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && mv bin /out" did not complete successfully: exit code: 2
Error: Process completed with exit code 1.

⚠️ As mentioned, it looks like it's using

a container builder
the docker.io/docker/dockerfile-upstream:master frontend (sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843)

#1 [internal] booting buildkit
#1 starting container buildx_buildkit_builder-9473aa25-e043-4654-934c-be4271ea8e3d0
#1 starting container buildx_buildkit_builder-9473aa25-e043-4654-934c-be4271ea8e3d0 0.6s done
#1 DONE 0.6s

#2 [internal] load .dockerignore
#2 transferring context: 56B done
#2 DONE 0.0s

#3 [internal] load build definition from Dockerfile
#3 transferring dockerfile: 11.57kB done
#3 DONE 0.0s

#4 resolve image config for docker.io/docker/dockerfile-upstream:master
#4 ...

#5 [auth] docker/dockerfile-upstream:pull token for registry-1.docker.io
#5 DONE 0.0s

#4 resolve image config for docker.io/docker/dockerfile-upstream:master
#4 DONE 0.8s

#6 docker-image://docker.io/docker/dockerfile-upstream:master@sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843
#6 resolve docker.io/docker/dockerfile-upstream:master@sha256:0e39c80bd858c49913c85dd3636aae0f6c8266ff1dcc24e5aeabf98e93579843 done
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 1.05MB / 11.47MB 0.2s
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 3.15MB / 11.47MB 0.3s
#6 sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 11.47MB / 11.47MB 0.5s done
#6 extracting sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc
#6 extracting sha256:0268775466e82aa89ffd3bd52916812c04ce4f8fb96af0e504edd8f997ffbcfc 0.2s done
#6 DONE 0.8s

The text was updated successfully, but these errors were encountered:

thaJeztah · 2022-12-02T09:06:20Z

/cc @crazy-max

thaJeztah · 2022-12-02T09:30:27Z

As CGO appears to be involved, this may also be a Go caching issue; I see --mount=target=/root/.cache,type=cache is used, and Go is known to have issues with caching if CGO is involved (sometimes getting a cache-hit where it shouldn't etc); if --mount=target=/root/.cache,type=cache is shared between stages, then it's possible it's using the wrong cache.

crazy-max · 2022-12-02T10:10:23Z

I'm able to repro with:

$ docker buildx build --target integration-tests https://github.com/moby/buildkit.git#4febae4f874bd8ef52dec30e988c8fe0bc96b3b9
...
------
 > [containerd 1/1] RUN --mount=from=containerd-src,src=/usr/src/containerd,readwrite --mount=target=/root/.cache,type=cache   git fetch origin   && git checkout -q "v1.6.2"   && make bin/containerd   && make bin/containerd-shim-runc-v2   && make bin/ctr   && mv bin /out:
#0 2.845 + bin/containerd
#0 3.980 # github.com/miekg/pkcs11
#0 3.980 cgo: malformed DWARF TagVariable entry
#0 3.985 make: *** [Makefile:230: bin/containerd] Error 2
------

I will take a closer look.

crazy-max · 2022-12-02T10:22:09Z

This looks to be related to the golang alpine image 1.18. Switching to 1.18.7 and it works now. Any idea @tianon? Maybe an issue with Alpine 3.17 since 1.18.7 alpine image is based on Alpine 3.16.

Edit: confirmed, works fine with golang:1.18-alpine3.16.

thaJeztah · 2022-12-02T10:25:30Z

I know alpine dropped LLVM on s390x (see docker/cli#3822), but that shouldn't be related I think?

crazy-max · 2022-12-02T10:31:18Z

It looks fine with golang:1.19-alpine on master branch though: https://github.com/moby/buildkit/blob/master/Dockerfile#L25. Switching to golang:1.18-alpine on master and same issue. There's smth fishy with this version 🤔.

crazy-max · 2022-12-02T10:45:01Z

Ok this change https://go-review.googlesource.com/c/go/+/406816 is not yet backported to 1.18 branch.

thaJeztah · 2022-12-02T10:52:13Z

Oh, interesting; I saw it was reported with go1.17, so assumed it was included in 1.18, but I see now it was marked for go1.19.

We should request a backport for that for go1.18

thaJeztah · 2022-12-02T10:58:43Z

@crazy-max could this one also be related?

cmd/cgo: Wrong types in compiler errors with clang 14 golang/go#53013

thaJeztah · 2022-12-02T11:00:27Z

Ah, that one was back ported I see golang/go@e727f41

So the procedure on their side looks to be to leave a backport request comment? golang/go#53013 (comment)

Can we make an "as minimal as possible" reproducer to post together with that comment?

crazy-max · 2022-12-02T11:17:16Z

So the procedure on their side looks to be to leave a backport request comment?

Done: golang/go#53000 (comment)

thaJeztah · 2022-12-02T11:18:53Z

So it looks like buildkit is using golang:1.18-alpine (no minor version for Go, and no specific alpine version); I see 1.18-alpine is now an alias for 1.18.8-alpine3.17, 1.18-alpine3.17, 1.18.8-alpine, 1.18-alpine, so updated to alpine 3.17.

With alpine 3.16:

docker run -it --rm golang:1.18.8-alpine3.16 sh -c 'apk add --quiet --no-cache clang && clang --version'
Alpine clang version 13.0.1
Target: aarch64-alpine-linux-musl
Thread model: posix
InstalledDir: /usr/bin

With alpine 3.17:

docker run -it --rm golang:1.18.8-alpine3.17 sh -c 'apk add --quiet --no-cache clang && clang --version'
Alpine clang version 15.0.6
Target: aarch64-alpine-linux-musl
Thread model: posix
InstalledDir: /usr/bin

crazy-max · 2022-12-02T11:20:32Z

Opened #44571 to fix CI in the meantime.

This allows us to pin to a specific version of Alpine, in case the golang:alpine image switches to a newer version, which may at times be incompatible, e.g. see moby/moby#44570 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

This allows us to pin to a specific version of Alpine, in case the golang:alpine image switches to a newer version, which may at times be incompatible, e.g. see moby/moby#44570 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 7b56eba) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

This allows us to pin to a specific version of Alpine, in case the golang:alpine image switches to a newer version, which may at times be incompatible, e.g. see moby/moby#44570 Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

This allows us to pin to a specific version of Alpine, in case the golang:alpine image switches to a newer version, which may at times be incompatible, e.g. see moby/moby#44570 Signed-off-by: Sebastiaan van Stijn <github@gone.nl> (cherry picked from commit 1b0d6fc) Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

thaJeztah added kind/bug Bugs are bugs. The cause may or may not be known at triage time so debugging may be needed. status/needs-attention Calls for a collective discussion during a review session area/testing labels Dec 2, 2022

crazy-max mentioned this issue Dec 2, 2022

ci(buildkit): update buildkit ref to fix issue with alpine image #44571

Merged

This was referenced Dec 2, 2022

cgo: malformed DWARF TagVariable entry golang/go#53000

Closed

cgo: malformed DWARF TagVariable entry [1.18 backport] golang/go#57044

Closed

This was referenced Dec 2, 2022

Dockerfile: update Alpine to 3.17 moby/buildkit#3349

Merged

[0.10] Dockerfile: update Alpine to 3.16 moby/buildkit#3350

Closed

[0.10] Dockerfile: enforce Alpine version for Go image moby/buildkit#3353

Merged

thaJeztah closed this as completed in #44571 Dec 2, 2022

thaJeztah mentioned this issue Dec 4, 2022

[20.10 backport] compose: bump version to v2.14.0 docker/docker-ce-packaging#790

Merged

This was referenced Dec 4, 2022

Dockerfile: add ALPINE_VERSION build-arg docker/cli#3896

Merged

[20.10 backport] Dockerfile: add ALPINE_VERSION build-arg docker/cli#3897

Merged

thaJeztah mentioned this issue Dec 8, 2022

[c8d] ci(buildkit): update buildkit ref to fix issue with alpine image rumpl/moby#123

Merged

crazy-max mentioned this issue Jun 29, 2023

dockerfile: update Alpine to 3.18 moby/buildkit#3976

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry" #44570

ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry" #44570

thaJeztah commented Dec 2, 2022 •

edited

thaJeztah commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022

crazy-max commented Dec 2, 2022 •

edited

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022 •

edited

crazy-max commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022

ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry" #44570

ci: buildkit CI is broken; "cgo: malformed DWARF TagVariable entry" #44570

Comments

thaJeztah commented Dec 2, 2022 • edited

Description

Reproduce

Expected behavior

docker version

docker info

Additional Info

thaJeztah commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022

crazy-max commented Dec 2, 2022 • edited

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022 • edited

crazy-max commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022

thaJeztah commented Dec 2, 2022

crazy-max commented Dec 2, 2022

thaJeztah commented Dec 2, 2022 •

edited

crazy-max commented Dec 2, 2022 •

edited

crazy-max commented Dec 2, 2022 •

edited