Duplicate image tags in local docker causing cache in LocalImagesCache

[ashwin-jacob]

Hello!

I have 5 spring batch tests I am running on a Jenkins pipeline. I am getting the following error only on ONE of the tests:

12:51:53  17 Mar 2020;17:51:41.498 [user:] [request:] [main] INFO 
12:51:53                  o.testcontainers.DockerClientFactory - Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
12:51:53          �[37m�[1mℹ︎ Checking the system...�[0m�[0m
12:51:53          �[32m✔ Docker version should be at least 1.6.0�[0m
12:51:53  17 Mar 2020;17:51:41.508 [user:] [request:] [main] DEBUG
12:51:53                  c.g.d.core.command.AbstrDockerCmd - Cmd: f6fc0588843eb76a458bdc25cdc942fed474945aa3cfe7b635220dd1935dbfaa,<null>,true,<null>,<null>,<null>,<null>,{df,-P},<null>,<null>,com.github.dockerjava.core.exec.ExecCreateCmdExec@5cbe877d
12:51:53  17 Mar 2020;17:51:41.644 [user:] [request:] [tc-okhttp-stream-2116511124] DEBUG
12:51:53                  c.g.d.c.c.ExecStartResultCallback - STDOUT: Filesystem           1024-blocks    Used Available Capacity Mounted on
12:51:53  overlay              314561516  62506464 252055052  20% /
12:51:53  tmpfs                    65536         0     65536   0% /dev
12:51:53  tmpfs                  8132940         0   8132940   0% /sys/fs/cgroup
12:51:53  shm                      65536         0     65536   0% /dev/shm
12:51:53  /dev/xvdg1           314561516  62506464 252055052  20% /etc/resolv.conf
12:51:53  /dev/xvdg1           314561516  62506464 252055052  20% /etc/hostname
12:51:53  /dev/xvdg1           314561516  62506464 252055052  20% /etc/hosts
12:51:53  tmpfs                  8132940     25132   8107808   0% /run/docker.sock
12:51:53  tmpfs                  8132940         0   8132940   0% /proc/acpi
12:51:53  tmpfs                    65536         0     65536   0% /proc/kcore
12:51:53  tmpfs                    65536         0     65536   0% /proc/keys
12:51:53  tmpfs                    65536         0     65536   0% /proc/timer_list
12:51:53  tmpfs                    65536         0     65536   0% /proc/timer_stats
12:51:53  tmpfs                    65536         0     65536   0% /proc/sched_debug
12:51:53  tmpfs                  8132940         0   8132940   0% /proc/scsi
12:51:53  tmpfs                  8132940         0   8132940   0% /sys/firmware
12:51:53          �[32m✔ Docker environment should have more than 2GB free disk space�[0m
12:51:53  17 Mar 2020;17:51:41.690 [user:] [request:] [main] DEBUG
12:51:53                  c.g.d.core.command.AbstrDockerCmd - Cmd: ListImagesCmdImpl[imageNameFilter=<null>,showAll=false,filters=com.github.dockerjava.core.util.FiltersBuilder@0,execution=com.github.dockerjava.core.exec.ListImagesCmdExec@3ecd267f]
12:51:53  [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.915 s <<< FAILURE! - in com.oranj.snowflake.account.VendorAccountTransactionSnowflakeConfigurationTest
12:51:53  [ERROR] com.oranj.snowflake.account.VendorAccountTransactionSnowflakeConfigurationTest  Time elapsed: 1.914 s  <<< ERROR!
12:51:53  org.testcontainers.containers.ContainerLaunchException: Container startup failed
12:51:53  Caused by: org.testcontainers.containers.ContainerFetchException: Can't get Docker image: RemoteDockerImage(imageNameFuture=java.util.concurrent.CompletableFuture@37ddb69a[Completed normally], imagePullPolicy=DefaultPullPolicy(), dockerClient=LazyDockerClient.INSTANCE)
12:51:53  Caused by: java.lang.IllegalStateException: Duplicate key ImageData(createdAt=1970-01-18T18:51:50.819Z)
12:51:53 

It seems like when the JVM terminates it is not properly cleaning up after itself. Thus, when it starts up again it is trying to start the container but it sees a duplicate container there.

Environment:

• Server Version: 19.03.1
• API Version: 1.40
• Operating System: CentOS Linux 7 (Core)
• Total Memory: 15884 MB

[lizardeye]

Hi, we faced the same issue recently. Cause of the problem was not the testcontainers configuration, but docker itself.
Cleaning up all the images and other content helped.

docker system prune -af

P.S. Cleaning only containers didn't help

[Choobz]

Hello,

We faced the same issue too. The problem seem to be in LocalImagesCache:

 Stream.of(repoTags).collect(Collectors.toMap(
                    DockerImageName::new,
                    it -> ImageData.from(image)
                ))

No distinct is made there and the Collectors.toMap then crash if there's duplicate in repoTags. Since you're not really supposed to have duplicate entries in the registry, that's why pruning get rid of the problem.
Would be great if it didn't crash Test-Containers.

I got rid of it by overriding the class definition in my application and adding .distinct().

[rnorth]

Damn, looks like we have a bug if this is causing us to crash. @Choobz, if you'd care to submit a PR I'd be very grateful!

[Choobz]

Sure, I'll get to it.

[francescopeloi]

crashing for me too, thanks too look at it @Choobz !