Unexpected lack of parallelization using WITH DOCKER

[jdewar]

I am seeing some unexpected serialization in an earthly build, and have made a gist of a simple example: https://gist.github.com/jdewar/991f2c91c6f1550c446d454fed0a57ad

The gist's Earthfile runs both a test with a docker container (+fast) and a test that takes a while to build an artifact, then uses the artifact to analyze the sourcecode (+slow). Both of these paths depend on dependencies (+deps) being available, but after that they should be able to run in parallel.

If I remove the 'WITH DOCKER' block, then I see the parallel paths (+lib, +artifact) outputting as expected after +deps. I found some docs that earthly's docker-in-docker may be needed, but adding DO github.com/earthly/lib+INSTALL_DIND did not seem to affect the result.

[vladaionescu]

Thanks for the reproduction code @jdewar! Very useful - I was able to run it and verify reproduction.

I was also able to simplify it to the following:

Earthfile

VERSION 0.6

FROM alpine:3.13
WORKDIR /work
DO github.com/earthly/lib+INSTALL_DIND

all:
    BUILD +slow
    BUILD +fast

artifact: 
    FROM +deps
    RUN echo "ARTIFACT"; sleep 10; echo "ARTIFACT done"
    RUN echo "test" >plts
    SAVE ARTIFACT plts

slow:
    FROM +deps
    COPY +artifact/plts plts
    RUN echo "SLOW"; sleep 10; echo "SLOW done"

lib:
    FROM +deps
    RUN echo "LIB"; sleep 10; echo "LIB done"

fast:
    FROM +lib
    COPY docker-compose.yml ./
    WITH DOCKER --compose docker-compose.yml
        RUN echo "DOCKER"; sleep 10; echo "DOCKER done"
    END

deps:
    RUN echo "DEPS"; sleep 10; echo "DEPS done"

docker-compose.yml

version: '3'
services:
  postgres:
    image: postgres
    ports:
      - "5432:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: postgres

It seems that replacing --compose with a simple --pull postgres makes the problem go away. I suspect this is related to the way we inspect the docker-compose.yml file (e.g. to extract information about which images to pull).