BUILD --auto-skip is caching failures

[alexcb]

What went wrong?

VERSION --build-auto-skip 0.8

foo:
  FROM alpine
  RUN false

test:
  BUILD --auto-skip +foo

on the first run, it fails:

earthly --auto-skip-db-path=test.db --auto-skip +test
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)

Streaming logs to https://cloud.earthly.dev/builds/5a406978-9c0a-4251-a98e-a6f70feec269

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

              logbus | Setting organization "" and project ""
               +test | --> FROM +base
               +test | --> BUILD +foo
                +foo | --> FROM +base
              alpine | --> Load metadata alpine linux/amd64
                +foo | --> FROM alpine
                +foo | [----------] 100% FROM alpine
                +foo | --> RUN false
                +foo | ERROR Earthfile line 5:2
                +foo |       The command
                +foo |           RUN false
                +foo |       did not complete successfully. Exit code 1

================================== ❌ FAILURE ===================================

                +foo *failed* | Repeating the failure error...
                +foo *failed* | --> RUN false
                +foo *failed* | [no output]
                +foo *failed* | ERROR Earthfile line 5:2
                +foo *failed* |       The command
                +foo *failed* |           RUN false
                +foo *failed* |       did not complete successfully. Exit code 1

however on the second run auto-skip is activated:

$ earthly --auto-skip-db-path=test.db --auto-skip +test
 Init 🚀
————————————————————————————————————————————————————————————————————————————————

           buildkitd | Found buildkit daemon as docker container (earthly-buildkitd)

Streaming logs to https://cloud.earthly.dev/builds/c1ea9f00-dcac-4906-a3da-a76eeeca2b51

 Build 🔧
————————————————————————————————————————————————————————————————————————————————

              logbus | Setting organization "" and project ""
               +test | --> FROM +base
           auto-skip | Target +foo (hash 41aa6c9fb9b83fa2461fb4c99a6ad612dd652c63) has already been run. Skipping.
skipping builder.go bf code due to GlobalWaitBlockFtr
              output | --> exporting outputs

 Push Summary ⏫ (disabled)
————————————————————————————————————————————————————————————————————————————————

To enable pushing use earthly --push

 Local Output Summary 🎁
————————————————————————————————————————————————————————————————————————————————

What should have happened?

the first run failed, therefore no auto-skip entry should have been recorded.

the second run should not invoke auto-skip, and it should re-run the same step that failed.

What earthly version?

earthly version prerelease 1b3bfd4 linux/amd64; Ubuntu 23.10 (Mantic Minotaur)

Buildkit Logs

No response

Other Helpful Information

No response

[alexcb]

the hash is inserted when this call is made:

earthly/earthfile2llb/interpreter.go

Line 1294 in 1b3bfd4

saveHashFn()

calling this here is incorrect, since the Build function doesn't actually perform a build (i.e. by calling forceExecution), but instead is simply for converting the BUILD command into the appropriate LLB.

This code has an additional bug where the platform type doesn't seem to be used in calculating the hash.

If the hash is platform agnostic (which might make sense since we need to be careful when pushing multi-platform images), it seems like we need to wait for all platforms to build (execute) correctly before saving the hash.

[alexcb]

Note that this can still be replicated with:

VERSION --build-auto-skip 0.8

foo:
  FROM alpine
  RUN echo "running under $(uname -m)" && if [ "$(uname -m)" = "x86_64" ]; then sleep 15 && exit 1; fi
  SAVE IMAGE mytest:latest

test:
  BUILD --auto-skip --platform=linux/arm64 +foo
  BUILD --auto-skip --platform=linux/amd64 +foo

and running:

rm -f test.db; ~/gh/earthly/earthly/build/linux/amd64/earthly prune --reset;  ~/gh/earthly/earthly/build/linux/amd64/earthly --auto-skip-db-path=test.db --auto-skip +test; ~/gh/earthly/earthly/build/linux/amd64/earthly --auto-skip-db-path=test.db --auto-skip +test;