Updates to loaded images don't invalidate cache for RUN commands in WITH DOCKER

[JBergles]

I'm not sure if this is a bug or an issue with my understanding of how caching works in earthly, so please bear with me.

Earthfile

VERSION 0.6

build:
  FROM alpine:latest
  COPY test.txt /test.txt

example:
  FROM earthly/dind:alpine
  WITH DOCKER --load built-image:latest=+build
    RUN docker run built-image:latest cat /test.txt
  END

Steps:

1. Run earthly -P +example for the included Earthfile
2. Change the contents of test.txt
3. Run earthly -P +example again

Expected results:

The RUN docker run built-image:latest cat /test.txt command is executed again

Observed results:

RUN step is reported as cached. If --no-cache is used with the RUN step, I can see that the updated container image is being used.

 earthly/dind:alpine | --> Load metadata linux/arm64
       alpine:latest | --> Load metadata linux/arm64
             context | --> local context .
              +build | --> FROM alpine:latest
              +build | [██████████] 100% resolve docker.io/library/alpine:latest@sha256:bc41182d7ef5ffc53a40b044e725193bc10142a1243f395ee852a8d9730fc2ad
             ongoing | context (5 seconds ago)
             context | [          ]   0% transferring .:
             context | sent 1 file stat)
             context | [██████████] 100% transferring .:
              +build | --> COPY test.txt /test.txt
              output | --> exporting outputs
              output | [██████████] 100% exporting layers
              output | [██████████] 100% exporting manifest sha256:4dd11cc29e9f8e60fc2965922107430e5140f5dfe78a1428e689bfeebac41314
              output | [██████████] 100% exporting config sha256:58fb16adcd4da2e90b60d06720473c24aa20d31ec12891988519544af0bea009
              output | [          ]   0% transferring docker.io/library/built-image:latest
Warning: Could not detect digest for image built-image:latest. Please update your buildkit installation.
            +example | --> FROM earthly/dind:alpine
            +example | [██████████] 100% resolve docker.io/earthly/dind:alpine@sha256:9b09804c68a2c68195b71e61a09616657b8f0a5ca01b4e029f710c7ca2396dc6
            +example | *cached* --> WITH DOCKER (install deps)
            +example | *cached* --> WITH DOCKER RUN --privileged docker run built-image:latest cat /test.txt
              output | [██████████] 100% transferring docker.io/library/built-image:latest

earthly-example.zip

[rrjjvv]

@JBergles I've reproduced this behavior; I'm assuming it's the same cause (so adding to your issue rather than creating a new one), but can you confirm whether you were using 0.6.26 or newer?

I believe this is specific to using the internal registry.

What I used as the basis for my reproduction:

VERSION 0.6

img1:
    FROM alpine
    RUN echo "41" > /num
    CMD ["cat", "/num"]

test1:
    FROM earthly/dind:alpine
    WITH DOCKER --load img=+img1
        RUN docker run --rm img && \
        [ "$(docker run --rm img)" -eq 41 ]
            
    END

With version 0.6.25 (where the default behavior uses tarballs):

1. First build (with --no-cache) behaves as expected and passes
2. Changing "41" -> "42" in the image results in the inner RUN executing and failing (a good thing)

With version 0.6.26 (where default behavior uses registry):

1. First build (with --no-cache) behaves as expected and passes
2. Changing "41" -> "42" in the image results in the inner RUN being skipped and 'passing' (a bad thing, since the image's "42" should fail the test's assertion for "41")

Using feature flags this behavior can be forced to flip-flop (strongly hinting that the registry functionality is responsible, not features/regressions in these specific versions). Specifically:

1. The default caching behavior in 0.6.26 is undesirable, but can be "fixed" by using VERSION --no-use-registry-for-with-docker 0.6
2. In reverse, the caching behavion in 0.6.25 is desirable, but using VERSION --use-registry-for-with-docker 0.6 will force the undesirable behavior.

I can post raw logs, but thought it would be more effective/useful to point out the potential culprit.

[JBergles]

@JBergles I've reproduced this behavior; I'm assuming it's the same cause (so adding to your issue rather than creating a new one), but can you confirm whether you were using 0.6.26 or newer?

Yes, I'm on version 0.6.28 7e4f1df

Using feature flags this behavior can be forced to flip-flop (strongly hinting that the registry functionality is responsible, not features/regressions in these specific versions). Specifically:

1. The default caching behavior in 0.6.26 is undesirable, but can be "fixed" by using VERSION --no-use-registry-for-with-docker 0.6
2. In reverse, the caching behavion in 0.6.25 is desirable, but using VERSION --use-registry-for-with-docker 0.6 will force the undesirable behavior.

--no-use-registry-for-with-docker does behave the way I expect