feat(elixir): use RUN caches

[thomaseizinger]

In my local testing using the docker-compose file, I had to rebuild the containers several times which always takes quite a while if the commands don't have access to a build cache.

I think there are probably more ways of optimising this Dockerfile. For example, using these caches it is usually not necessary to compile the dependencies separately from the apps because it behaves as an incremental build without docker if anything changes.

I don't know the ins and outs of elixir though so I didn't want to touch that bit but very likely, this can probably just be combined into a single(?) RUN step that downloads dependencies and compiles the requested app.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Updated (UTC)
firezone	⬜️ Ignored (Inspect)		Sep 27, 2023 4:28am

[thomaseizinger]

This bit is important, we need to cp the final artifacts out of the cache, otherwise the next container can't access them.

[thomaseizinger]

@AndrewDryga I saw that you have some volumes going on as well to cache build artifacts. This PR likely is not sufficient to replace all of that but it could be a direction worthwhile exploring to simplify the cache setup.

[jamilbk]

Thank you for adding this! FWIW the elixir devs usually just run the postgres service with docker-compose and then start the vanilla Elixir services with mix phx.server from the elixir/ directory. This is because we're on macOS where there's a non-trivial performance hit for going across the filesystem barrier though.

[AndrewDryga]

Just realized that it's not a good idea to do so.

We can build containers for different architectures and the caches should not be reused between them - it will affect our production containers too and we don't want them to pick up whatever is left after compiling tests on CI.

This will also be problematic if the Elixir version within docker is not exactly the same as on the host (eg. MacOS can lag behind due to some arm bugs).

So having cache for builds is nice but we definitely don't want to pick up the host cache.

[jamilbk]

@AndrewDryga D'oh, you're completely right, I forget that the Elixir _build dir is not architecture-segregated like cargo. @thomaseizinger I'm not sure there's a better approach other than letting Docker cache these layers implicitly.

[thomaseizinger]

I think there are some misunderstandings here :)

1. These RUN caches are managed by docker and are effectively volumes stored in /var/lib/docker/.... They don't map to the equivalent directory on the host machine.
2. I think docker separates these accordingly per architecture. If you are building for example a docker ARM image vs a docker x64 image.
3. Docker volumes are not cached in CI so this cache doesn't work there anyway.

The only thing these instructions do is have an implicit volume that is mounted for the RUN step. In other words, if you rebuild the an image, the image is going to find its previous build artifacts and thus complete the RUN step quicker.

[thomaseizinger]

2. I think docker separates these accordingly per architecture. If you are building for example a docker ARM image vs a docker x64 image.

Okay, it doesn't but we can make our own key that uniquely identifies a cache location that should be shared: https://docs.docker.com/engine/reference/builder/#run---mounttypecache

[jamilbk]

@thomaseizinger Apologies, been jumping around a bit too much lately and didn't give this the proper review. I'll defer to @AndrewDryga to do the final pass. In general would be in favor of caching.

From the docs link you posted it seems like this is the designated purpose for these build caches.

[AndrewDryga]

@thomaseizinger if we can split the cache for different architectures for issue (2) - we should be safe 👍 .

[thomaseizinger]

@thomaseizinger if we can split the cache for different architectures for issue (2) - we should be safe 👍 .

Okay, I'll have a play with it! Should I also attempt to combine the different RUN steps where we currently compile dependencies first etc? Also, I saw that the docker-compose file has certain volumes mounted. Are those only for build caches too?

[AndrewDryga]

@thomaseizinger Those volumes are mounted for a container where Elixir code is not going through the release cycle, which is intended to be a local debugging tool. In theory it will allow you to change files on host and live-reload code inside of the container but I never tested it.

Should I also attempt to combine the different RUN steps where we currently compile dependencies first etc?

Those steps are separate intentionally to minimize builder state changes, eg. allowing reusing fetched and compiled deps even if the application code changed.

[thomaseizinger]

Should I also attempt to combine the different RUN steps where we currently compile dependencies first etc?

Those steps are separate intentionally to minimize builder state changes, eg. allowing reusing fetched and compiled deps even if the application code changed.

Right, I figured that. Using a RUN cache serves the same purpose. It really is just like incremental builds locally which allows you to remove all this complexity from the dockerfile.

[thomaseizinger]

Closing this because docker does not support caching caches 🙃

Related: docker/build-push-action#743.