Some operations are hindering parallelism

[flupke]

It seems that FROM DOCKERFILE creates synchronisation points in the build graph:

FROM alpine:latest

a:
    RUN echo -e "FROM alpine:latest\nRUN sleep 4\n" > Dockerfile
    RUN sleep 5
    SAVE ARTIFACT Dockerfile

b:
    RUN echo -e "FROM alpine:latest\nRUN sleep 5\n" > Dockerfile
    RUN sleep 5
    SAVE ARTIFACT Dockerfile

a-child:
    FROM DOCKERFILE +a/
    RUN echo "A done"

b-child:
    FROM DOCKERFILE +b/
    RUN echo "B done"

all:
    BUILD +a-child
    BUILD +b-child

Here is what happens when trying to build this:

Building a similar graph without FROM DOCKERFILE or with locally referenced Dockerfile executes +a and +b in parallel.

Note: I had to do earthly prune because --no-cache didn't prevent the *cached* steps, is this normal? (using earthly v0.5.7)

[vladaionescu]

Hi @flupke - you're right. This is a limitation in our current implementation. It affects WITH DOCKER --load, WITH DOCKER --pull, FROM DOCKERFILE <target>, IF and non-constant args: ARG X=$(...) and --build-arg X=$(...).

Technical explanation: buildkit takes care of parallelism for the most part, however, some Earthfile operations need to inquire about the result of a previous operation before being able to continue. This takes place in the Earthfile->LLB conversion process. Currently, this conversion process is single-threaded, so situations like you're demoing here make this bottleneck obvious.

We've been thinking of switching the conversion process to be parallel too but haven't yet gotten around to it. I'll use this issue as a tracking issue.

Note: I had to do earthly prune because --no-cache didn't prevent the cached steps, is this normal? (using earthly v0.5.7)

I believe this is just a visual glitch. Mind opening a separate issue for this?

[flupke]

I believe this is just a visual glitch. Mind opening a separate issue for this?

Sure! #889

BTW thank you very much for the awesome support and product, Earthly is amazing ❤️

[vladaionescu]

Note that there are some known issues with this feature:

• Random context deadline exceeded returned by buildkit for large builds, or for moderately-sized builds running on underpowered or virtualized setups (e.g. Mac, Windows).

Will keep this open until known issues are addressed.

[vladaionescu]

To enable this Beta feature set export EARTHLY_CONVERSION_PARALLELISM=5 before running Earthly.