ONBUILD instructions in earlier stages of multi-stage build do not run before COPY --from in later stages

[nightlark]

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Building a docker image fails when the FROM image for the final stage uses a [ONBUILD] COPY --from to copy a directory or file from an earlier stage, if the earlier named stage has an ONBUILD instruction that needs to run before the source of directory or file is created.

If the source file or directory to copy doesn't exist, the error given is some variant of: ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 33a75f97-2e2d-44b6-be3b-0c3fc217ec59::onn3gmwfm28dr2bgekr7461yy: "/build/dist": not found

However, the image build can also appear to succeed but not have the expected final state -- if the directory/file already exists in the stage being copied from it will copy what is there, however an ONBUILD instructions for the earlier stage may modify the file or add files to an empty directory. That updated file/directory does not get copied as would be expected.

Expected behaviour

I would expect that if the earlier stage contains ONBUILD instructions, that those would run before trying to copy contents from the earlier stages into a later stage, to avoid unexpected results.

Actual behaviour

The actual behavior is that the directive for copying a directory/file runs prior to the ONBUILD instructions for earlier stages. This has results that range from an error if the source to copy is created by the ONBUILD instruction, or (worse because its harder to catch) copying outdated file/directory contents that are supposed to be updated by the ONBUILD instructions.

Buildx version

github.com/docker/buildx v0.11.2 9872040

Docker info

Client: Docker Engine - Community
 Version:    24.0.5
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.20.2
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scan: Docker Scan (Docker Inc.)
    Version:  v0.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-scan

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 21
 Server Version: 24.0.5
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8165feabfdfe38c65b599c4993d227328c231fca
 runc version: v1.1.8-0-g82f18fe
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 4.18.0-513.11.1.el8_9.x86_64
 Operating System: Red Hat Enterprise Linux 8.9 (Ootpa)
 OSType: linux
 Architecture: x86_64
 CPUs: 20
 Total Memory: 62.32GiB
 Name: <redacted>
 ID: 33a75f97-2e2d-44b6-be3b-0c3fc217ec59
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT             PLATFORMS
default * docker                                       
  default default         running v0.11.6+0a15675913b7 linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/amd64/v4, linux/386

Configuration

my-builder Dockerfile:

FROM python:3.11.7-slim

ONBUILD RUN mkdir -p build/dist/
ONBUILD RUN touch build/dist/a_file

Build with: sudo docker build -t my-builder:latest .

my-base Dockerfile:

FROM python:3.11.7-slim

ONBUILD COPY --from=builder /build/dist/ /root/dist

Build with: sudo docker build -t my-base:latest .

Multi-stage build Dockerfile:

FROM my-builder:latest as builder

FROM my-base:latest

Build with: sudo docker build --no-cache .

This should result in an error along the lines of "failed to computer cache key". However, if in the my-builder Dockerfile the ONBUILD prefix is removed from the line that creates that directory (but leave it on the 2nd RUN instruction!), then we can see that building this multi-stage Docker image appears to succeed. However, when we go to look at the contents of the image, /root/dist does not contain the a_file that gets created as part of the ONBUILD instruction in the earlier builder stage.

Build logs

[+] Building 0.1s (6/6) FINISHED                                                                                                                                                                                                      docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                                                            0.0s
 => => transferring dockerfile: 137B                                                                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                               0.0s
 => => transferring context: 2B                                                                                                                                                                                                                 0.0s
 => [internal] load metadata for docker.io/library/my-base:latest                                                                                                                                                                               0.0s
 => [builder 1/1] FROM docker.io/library/my-builder:latest                                                                                                                                                                                      0.0s
 => CACHED [stage-1 1/1] FROM docker.io/library/my-base:latest                                                                                                                                                                                  0.0s
 => ERROR [stage-1 2/1] COPY --from=builder /build/dist/ /root/dist                                                                                                                                                                             0.0s
------
 > [stage-1 2/1] COPY --from=builder /build/dist/ /root/dist:
------
Dockerfile:1
--------------------
   1 | >>> FROM my-builder:latest as builder
   2 |     
   3 |     FROM my-base:latest
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref 33a75f97-2e2d-44b6-be3b-0c3fc217ec59::a13omyrr4f5v0dbdmebku3tt6: failed to walk /var/lib/docker/tmp/buildkit-mount3383600270/build: lstat /var/lib/docker/tmp/buildkit-mount3383600270/build: no such file or directory

Additional info

No response

[dvdksn]

ONBUILD COPY --from is not supported.

[nightlark]

Even just regular COPY without the ONBUILD fails to work — it either copies a stale file/folder from the earlier stage containing ONBUILD instructions, or it fails if the file/folder doesn’t exist before the ONBUILD instructions in the earlier stage are run.

Revisiting this after some rest, it actually looks like inserting a regular COPY (not ONBUILD) in the final stage after the FROM command does work, and the ONBUILD instructions from the earlier stage get evaluated first. From the docs (https://docs.docker.com/reference/dockerfile/#onbuild), The trigger will be executed in the context of the downstream build, as if it had been inserted immediately after the FROM instruction in the downstream Dockerfile implies that ONBUILD COPY should have identical behavior.