'failed to compute cache key' error with invalid symbolic link

[dvic]

failed to compute cache key: "/src/deps/oban_web/lib/oban_web/lib/oban_web" not found: not found

This is the tree, as you can see it contains an invalid symlink.

        +elixir-deps | deps/oban_web
        +elixir-deps | ├── hex_metadata.config
        +elixir-deps | ├── lib
        +elixir-deps | │   └── oban
        +elixir-deps | │       ├── web
        +elixir-deps | │       │   ├── components
        +elixir-deps | │       │   │   ├── bulk_action_component.ex
        +elixir-deps | │       │   │   ├── detail_component.ex
        +elixir-deps | │       │   │   ├── header_component.ex
        +elixir-deps | │       │   │   ├── listing_component.ex
        +elixir-deps | │       │   │   ├── listing_row_component.ex
        +elixir-deps | │       │   │   ├── node_component.ex
        +elixir-deps | │       │   │   ├── notification_component.ex
        +elixir-deps | │       │   │   ├── queue_component.ex
        +elixir-deps | │       │   │   ├── refresh_component.ex
        +elixir-deps | │       │   │   ├── search_component.ex
        +elixir-deps | │       │   │   ├── sidebar_component.ex
        +elixir-deps | │       │   │   ├── state_component.ex
        +elixir-deps | │       │   │   └── timeline_component.ex
        +elixir-deps | │       │   ├── helpers.ex
        +elixir-deps | │       │   ├── live
        +elixir-deps | │       │   │   └── dashboard_live.ex
        +elixir-deps | │       │   ├── oban_web -> ../../oban_web/lib/oban_web

Is this a docker bug or an earthly bug? I'm not sure what the correct behavior should be here, throw a more clear error?

[alexcb]

This might possibly be related to #1062 in which the file is being accessed without being explicitly referenced?

We can most definitely improve the error message here.

We do have an experimental COPY --if-exists flag which only performs a copy if the file exists, perhaps this already addresses downgrading the error into a warning? And if not, maybe that's something we should do?

[dvic]

In our case we got this error with the following line: SAVE ARTIFACT deps/*. I think improving the error message here is the easiest thing to do, although it would be even better if the broken symlink would just work and get saved as-is.

[jamesalucas]

I am seeing something similar. This Earthfile works with Earthly v0.5.16 but not with v0.5.17:

nodejs:
    FROM node:14.15.4-alpine3.12
    RUN rm -rf /usr/local/lib/node_modules /usr/local/bin/npm
    SAVE ARTIFACT /usr/ /usr/
    SAVE ARTIFACT /opt/ /opt/

build-image:
    FROM alpine:3.12.7
    COPY +nodejs/* /
    WORKDIR /app

/usr/local/bin/earthly-0.5.17 --no-cache +build-image
           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
       alpine:3.12.7 | --> Load metadata linux/amd64
node:14.15.4-alpine3.12 | --> Load metadata linux/amd64
             +nodejs | --> FROM node:14.15.4-alpine3.12
        +build-image | --> FROM alpine:3.12.7
             +nodejs | [██████████] resolve docker.io/library/node:14.15.4-alpine3.12@sha256:55bf28ea11b18fd914e1242835ea3299ec76f5a034e8c6e42b2ede70064e338c ... 100%
        +build-image | [██████████] resolve docker.io/library/alpine:3.12.7@sha256:87703314048c40236c6d674424159ee862e2b96ce1c37c62d877e21ed27a387e ... 100%
             +nodejs | --> RUN rm -rf /usr/local/lib/node_modules /usr/local/bin/npm
             +nodejs | --> SAVE ARTIFACT /usr/ +nodejs/usr/usr
             +nodejs | --> SAVE ARTIFACT /opt/ +nodejs/opt/opt
             +nodejs | --> SAVE ARTIFACT /bin/ +nodejs/bin/bin
        +build-image | --> COPY +nodejs/* /
        +build-image | WARN: (COPY +nodejs/* /) "/bin/busybox" not found: not found
Error: failed to compute cache key: "/bin/busybox" not found: not found

/usr/local/bin/earthly-0.5.16 --no-cache +build-image
           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
           buildkitd | Updated image available. Restarting buildkit daemon...
           buildkitd | ...Done
       alpine:3.12.7 | --> Load metadata linux/amd64
node:14.15.4-alpine3.12 | --> Load metadata linux/amd64
             +nodejs | --> FROM node:14.15.4-alpine3.12
        +build-image | --> FROM alpine:3.12.7
             +nodejs | [██████████] resolve docker.io/library/node:14.15.4-alpine3.12@sha256:55bf28ea11b18fd914e1242835ea3299ec76f5a034e8c6e42b2ede70064e338c ... 100%
        +build-image | [██████████] resolve docker.io/library/alpine:3.12.7@sha256:87703314048c40236c6d674424159ee862e2b96ce1c37c62d877e21ed27a387e ... 100%
             +nodejs | --> RUN rm -rf /usr/local/lib/node_modules /usr/local/bin/npm
             +nodejs | --> SAVE ARTIFACT /usr/ +nodejs/usr/usr
             +nodejs | --> SAVE ARTIFACT /opt/ +nodejs/opt/opt
        +build-image | --> COPY +nodejs/* /
        +build-image | --> WORKDIR /app
              output | --> exporting outputs
=================================== SUCCESS ====================================

I think it's because many of the files that have been saved as artifacts are symbolic links to /bin/busybox and they would work once copied into the target image as it is also Alpine based and so /bin/busybox would exist.

[janwillemvd]

@jamesalucas I encountered the same yesterday. v0.5.15 works, v0.5.17 does not.

[alexcb]

@jamesalucas I too am getting the same error; thanks for reproducing the bug, that'll make it easier for us to fix.

[alexcb]

under v0.5.15, earthly is resolving symlinks when copying a specific file:

symlinked-file:
    RUN mkdir /root/datastore
    RUN echo -n thedata > /root/datastore/data.txt
    RUN mkdir /root/symlinks
    RUN ln -s /root/datastore/data.txt /root/symlinks/datalink
    SAVE ARTIFACT /root/symlinks/datalink

copy-symlinked-file:
    COPY +symlinked-file/datalink .
    RUN test $(cat datalink) = "thedata"

In this case the datalink file under the copy-symlinked-file will end up as a regular file.

However when copying a directory containing a symlink, earthly is not copying the contents of the file like in the above example, but instead maintains the symlinks, e.g.

dir-containing-symlinked-file:
    RUN mkdir /root/datastore
    RUN echo -n thedata > /root/datastore/data.txt
    RUN mkdir /root/symlinks
    RUN ln -s /root/datastore/data.txt /root/symlinks/datalink
    SAVE ARTIFACT /root/symlinks

copy-dir-containing-symlinked-file:
    RUN mkdir /root/datastore
    RUN echo -n overridden-data > /root/datastore/data.txt
    COPY +dir-containing-symlinked-file/symlinks symlinks
    RUN test $(cat symlinks/datalink) = "overridden-data"

This used to work under v0.5.15; however, something changed in v0.5.17 and I am still looking into a fix.

[alexcb]

narrowing this down a bit more, this builds:

FROM alpine:3.13

a:
    SAVE ARTIFACT /usr/ /usr/

b:
    COPY +a/usr /

output:

$ earthly-v0.5.17 --no-cache /tmp/junk-symlink-4+b
           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
         alpine:3.13 | --> Load metadata linux/amd64
/t/junk-symlink-4+base | --> FROM alpine:3.13
/t/junk-symlink-4+base | [██████████] resolve docker.io/library/alpine:3.13@sha256:f51ff2d96627690d62fee79e6eecd9fa87429a38142b5df8a3bfbb26061df7fc ... 100%
 /t/junk-symlink-4+a | --> SAVE ARTIFACT /usr/ /tmp/junk-symlink-4+a/usr/usr
 /t/junk-symlink-4+b | --> COPY /tmp/junk-symlink-4+a/usr /
              output | --> exporting outputs
=================================== SUCCESS ====================================

HOWEVER, changing it to this:

FROM alpine:3.13

a:
    SAVE ARTIFACT /usr/ /usr/

b:
    COPY +a/* /

fails:

$ earthly-v0.5.17 --no-cache /tmp/junk-symlink-4+b
           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)
         alpine:3.13 | --> Load metadata linux/amd64
/t/junk-symlink-4+base | --> FROM alpine:3.13
/t/junk-symlink-4+base | [██████████] resolve docker.io/library/alpine:3.13@sha256:f51ff2d96627690d62fee79e6eecd9fa87429a38142b5df8a3bfbb26061df7fc ... 100%
 /t/junk-symlink-4+a | --> SAVE ARTIFACT /usr/ /tmp/junk-symlink-4+a/usr/usr
 /t/junk-symlink-4+b | --> COPY /tmp/junk-symlink-4+a/* /
 /t/junk-symlink-4+b | WARN: (COPY /tmp/junk-symlink-4+a/* /) "/bin/busybox" not found: not found
Error: failed to compute cache key: "/bin/busybox" not found: not found

[alexcb]

I'm still working at the issue. I noticed a similar problem exists in docker; consider a Dockerfile such as:

# syntax=docker/dockerfile:1
FROM alpine:3.13
RUN mkdir /root/datastore
RUN echo -n thedata > /root/datastore/data.txt
RUN mkdir /root/symlinks
RUN ln -s /root/datastore/data.txt /root/symlinks/datalink
RUN rm /root/datastore/data.txt


FROM alpine:3.13
WORKDIR /root
COPY --from=0 /root/symlinks/* .
RUN ls -la

this outputs the same error:

$ DOCKER_BUILDKIT=1 docker build .
[+] Building 1.3s (13/14)                                                                                                                                                                                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                                                                                                                                                                                          0.0s
 => => transferring dockerfile: 357B                                                                                                                                                                                                                                                                                                                          0.0s
 => [internal] load .dockerignore                                                                                                                                                                                                                                                                                                                             0.0s
 => => transferring context: 2B                                                                                                                                                                                                                                                                                                                               0.0s
 => resolve image config for docker.io/docker/dockerfile:1                                                                                                                                                                                                                                                                                                    0.4s
 => CACHED docker-image://docker.io/docker/dockerfile:1@sha256:e2a8561e419ab1ba6b2fe6cbdf49fd92b95912df1cf7d313c3e2230a333fdbcc                                                                                                                                                                                                                               0.0s
 => [internal] load metadata for docker.io/library/alpine:3.13                                                                                                                                                                                                                                                                                                0.0s
 => [stage-0 1/6] FROM docker.io/library/alpine:3.13                                                                                                                                                                                                                                                                                                          0.0s
 => CACHED [stage-0 2/6] RUN mkdir /root/datastore                                                                                                                                                                                                                                                                                                            0.0s
 => CACHED [stage-0 3/6] RUN echo -n thedata > /root/datastore/data.txt                                                                                                                                                                                                                                                                                       0.0s
 => CACHED [stage-0 4/6] RUN mkdir /root/symlinks                                                                                                                                                                                                                                                                                                             0.0s
 => CACHED [stage-0 5/6] RUN ln -s /root/datastore/data.txt /root/symlinks/datalink                                                                                                                                                                                                                                                                           0.0s
 => CACHED [stage-0 6/6] RUN rm /root/datastore/data.txt                                                                                                                                                                                                                                                                                                      0.0s
 => CACHED [stage-1 2/4] WORKDIR /root                                                                                                                                                                                                                                                                                                                        0.0s
 => ERROR [stage-1 3/4] COPY --from=0 /root/symlinks/* .                                                                                                                                                                                                                                                                                                      0.0s
------
 > [stage-1 3/4] COPY --from=0 /root/symlinks/* .:
------
failed to compute cache key: "/root/datastore/data.txt" not found: not found

and this makes a bit of sense, as the file was removed after I linked it, but before the second target could copy it.

I'm still leaning towards restoring our old functionality where we just made a symlink even if it doesn't point to a valid file.

[dvic]

I'm still leaning towards restoring our old functionality where we just made a symlink even if it doesn't point to a valid file.

I also think this is also the way forward, it should be possible to copy a broken symlink imho.

[alexcb]

We have released https://github.com/earthly/earthly/releases/tag/v0.5.18 which contains the new --symlink-no-follow flag which can be passed to the COPY (or SAVE ARTIFACT). This should make it possible to copy invalid symlinks.