Approach for reusing intermediate Docker images in system tests

[MakisH]

In the preCICE v2 paper, we describe the challenges, stakeholders, and general approach for the preCICE system tests, which test together multiple repositories, built as layers of Docker images on top of each other, and started as different services. This is something we already tried in the previous, Travis-based system tests.

To save significant time, we want to avoid rebuilding images of components that could already be available. In the previous approach, we were using a rather arbitrary naming scheme for images that we were pushing to Docker hub. Most of the images are not available anymore, but they were quite a few.

In the new system tests, I can think of the following options, which need further investigation:

1. Use again a naming scheme and push to the GitHub Container Registry.
   • Pro: Rather straight-forward
   • Contra: Difficult to come up with a naming scheme that covers everything we need, and tricky to enforce it
   • Contra: We will again end up with several images that are not meant to be public.
   • Contra: We will have to find out and manage which images are still needed.
2. Provide some storage server and configure Docker to use it for building the images. As Docker builds images in layers, we should be able to reuse the existing system.
   • Pro: No need to define a naming scheme: Docker handles this internally, with hashes.
   • Pro: Not exposing images that are not meant to be used by users
   • Pro: We could reuse the Docker tools to find out which layers are not needed anymore, and prune them.
   • Pro: We avoid interacting between repositories just to push/pull intermediate images.
   • Contra: No idea how to configure this, but this seems to be a starting point
   • Contra: We need to set up some dedicated space. But we have options.

Alternatively, or next to option 2, we could provide only one base image (e.g., the official preCICE image) and use Terraform (or maybe Ansible) to generate the images we need at each job. We will again need some way to reuse artifacts.

Are there any further alternatives we should consider? Any suggestions on the technical implementation?

[valentin-seitz]

About the second option:
Docker apparently has some shared cache storage mechanism built in.
Using this and also converting the Dockerfiles to multistage builds i can see a path where we could be able to build images on the fly without any big overhead.
One problem that might arise is the location of the storage, as dockers buildx currently doesnt support many backends.

[MakisH]

One problem that might arise is the location of the storage, as dockers buildx currently doesnt support many backends.

Interestingly, one of the supported backends is also the GitHub Actions Cache.

We can use different/multiple caches, but we need to name each of them differently. If we also set the cache mode to max, this will cache all the intermediate layers, increasing the chances for cache hits.

[valentin-seitz]

With the current approach of having N Dockerfiles for N containers used during the systemtests this makes the Dockerfiles become quite spread out and loosely coupled.
With this loose coupling we would need a way to enforce some common practices among them (do we create a precice user with uid=1000 or not, etc.).

A very different approach might be to just create one huge Dockerfile and just use multiple stages to build the adapters.

This could look like:

FROM ubuntu:22_04 as precice_base
ARG PRECICE_REF
RUN update_os()
RUN ./install_precice ${PRECICE_REF}

# Build openfoam-adapter
FROM precice_base as openfoam-adapter
ARG OPENFOAM_ADAPTER_REF
RUN ./Allwmake ${OPENFOAM_ADAPTER_REF}


# Build python_bindings image
FROM precice_base as python_bindings
ARG PYTHON_BINDINGS_REF
RUN ./install_python_bindings ${PYTHON_BINDINGS_REF}

# Solve dependency of fenics on python bindings like.
FROM python_bindings as fenics-adapter
ARG FENICS_ADAPTER_REF
RUN ./install_fenics_adapter ${FENICS_ADAPTER_REF}

One could then use the --target clause in the docker build(x) command to build exactly the container you want like: docker build --target openfoam-adapter

This approach has of course Pros and Cons:

Pro:

• Will also allow precice/precice to be rebuilt without needing to publish a docker container before all adapter can rebuilt on that base image
• the base image is only pulled instead of N times where N is the number of coupling participants.
• Allows for easy standardization of the docker containers.
• parallel builds of independent stages (with buildx i think)
• just supply the build-arguments of your dreams and docker will run and built that image for you (no relying on a tag beeing published etc)
• should hit the cache if configured correctly

Cons

• one chunky Dockerfile
• where to put the Dockerfile
• Probably repo hopping is needed if something changes to the way things are built
• maybe need for additional containers for releasing packages etc. (double the maintenance effort)

[MakisH]

I understand that this is the natural implementation of what we have established so far. I would like to see a working prototype.

should hit the cache if configured correctly

This would be the main thing to check, and what could actually also speed up everything.

where to put the Dockerfile

In the precice/tutorials repository, probably in the tools/tests directory (or just tools/, if we want to reuse it outside the tests context). Keeping it in this repository would also make local builds easier.

Probably repo hopping is needed if something changes to the way things are built

I do not understand this. Could you please elaborate?

maybe need for additional containers for releasing packages etc. (double the maintenance effort)

I guess this means "what happens if we also want to publish Docker images for each repository". I would say, let's ignore that for now. Rethinking about it, it would not really be something that I would like to support users with.

[valentin-seitz]

Probably repo hopping is needed if something changes to the way things are built

What i mean by that: Assume someone changes the way the openfoam-adapter is beeing built. Like having additional compiler flags or what not. Then one would not only need to touch the openfoam-adapter repository, but also the tutorials tutorial to update the respective Dockerfile.
This removes some locality we would have with a "Everbody does its own dockerfile in their own repo" approach

I guess this means "what happens if we also want to publish Docker images for each repository". I would say, let's ignore that for now. Rethinking about it, it would not really be something that I would like to support users with.

Thinking about it: If done right one could just easily reuse the dockerfile do publish an image to Dockerhub or any other registry by building the specific stage. This could easily be done by the build-and-push action

I understand that this is the natural implementation of what we have established so far. I would like to see a working prototype.

So should i go ahead and try to build a small prototype using the openfoam-adapter?
We could use this to benchmark building speeds a bit?

[MakisH]

Probably repo hopping is needed if something changes to the way things are built

What i mean by that: Assume someone changes the way the openfoam-adapter is beeing built. Like having additional compiler flags or what not. Then one would not only need to touch the openfoam-adapter repository, but also the tutorials tutorial to update the respective Dockerfile. This removes some locality we would have with a "Everbody does its own dockerfile in their own repo" approach

I think we should be able to find a solution to that later, if such a problem becomes realistic.

One approach could be, e.g., to have the templates of the Docker files etc in each repository. But these are also specific to the platform used, so I think it makes most sense to keep them where the tests are. We also do this in the precice/vm repo.

I understand that this is the natural implementation of what we have established so far. I would like to see a working prototype.

So should i go ahead and try to build a small prototype using the openfoam-adapter? We could use this to benchmark building speeds a bit?

Yes, please! 🚲

[MakisH]

Follow-up: I understand that the idea now is that we use multi-stage Docker builds just to construct the containers that we will be using in Docker Compose. Is that correct?

Or do we now want to have a single container with all the components tested in each test?

I would guess that a combination would be faster, as it would not need to generate new images that often: use multi-stage builds to take advantage of the caching when switching between states of each image for each component, but then still use Docker Compose to reuse images of components that have not been changed.

So, more like this:

* preCICE ref           * preCICE ref           
|                       |
|                       * Python bindings ref
|                       |
* OF adapter ref        * FEniCS adapter ref
|                       |
\                       /
 \______ compose ______/

rather than:

* preCICE ref
|
* Python bindings ref
|
* FEniCS adapter ref
|
* OpenFOAM adapter ref
|
single image

[valentin-seitz]

I would go with:

* preCICE ref           * preCICE ref           
|                       |
|                       * Python bindings ref
|                       |
* OF adapter ref        * FEniCS adapter ref
|                       |
\                       /
 \______ compose ______/

I also did some experimenting. The flexibility of not having to rely on a published tag is quite nice.
Its quite important to order your layers smart in order for this approach to be fast, so maybe we should at some point do some experimenting which ordering and caching mechanism is the fastest for us. [

[fsimonis]

@valentin-seitz can you provide some comment on why this issue was closed? A reference to a pull request or other issue are also great.

[MakisH]

These are the related PRs:

1. Have one Dockerfile instead of multiples  tutorials#364
2. Add test workflows  tutorials#372

The tools/tests/dockerfiles/ubuntu_2204/Dockerfile uses a multi-stage build, which works locally, independently of the GitHub Actions.

The Docker Compose template then configures the GitHub Actions cache: https://github.com/precice/tutorials/blob/develop/tools/tests/docker-compose.template.yaml

Cached layers are visible here (when available): https://github.com/precice/tutorials/actions/caches

While the GitHUb Actions cache is updated, it is not hit. There seems to be a related bug in the Action. If we switched the system tests to our own runner, we could just rely on the normal Docker cache anyway.