[RFC] Add ability to cache docker images

[jakeleventhal]

Which project is this feature idea for?

Turborepo

Describe the feature you'd like to request

Turbo should be able to cache docker images and use those with cache hits.

Describe the solution you'd like

Implement caching with docker images.

Describe alternatives you've considered

N/A

[chris-olszewski]

Hi Jake, could you elaborate a little bit more on what you mean by having Turborepo cache docker images? Docker itself already has fairly advanced image caching.

[jakeleventhal]

sure, what im referring to is being able to cache the output of a docker build much in the same way that i would cache any other output in turbo. For instance, i should be able to run a script "image" via turbo image which builds a docker image, and specify the image as an output (similar to how bazel works). then in ci, my remote cache should be able to recognize a cache hit, and download the docker artifact as an output

[mehulkar]

@jakeleventhal what happens when you do that today? I imagine the artifact size is pretty large, so writing to cache (especially remote cache) could take a while, but turbo shouldn't have an opinion over what the outputs are. If you're running into issues, can you provide a reproduction repo?

[jakeleventhal]

how exactly would you even specify a docker image output?

[mehulkar]

sorry, it's been ~a decade since I used docker, does it not create the image on your local disk first? Is it no longer a local directory or file on your filesystem when you create the image?

[jakeleventhal]

They get saved to a ~/.docker directory

[mehulkar]

Is it possible to customize that location such that you could put it in the repo location? (I assume that even if that is possible it may interfere with other docker operations). That said, it seems like there are a couple general purpose features here:

• outputs outside the repo root (which is already possible, I think)
• or maybe a --cache-dir per task, so in your case the image task could specify the location

Will bring this up with the team!

[jakeleventhal]

There isn't a way to specify this unfortunately.

[anthonyshew]

We just discussed this amongst the team and we figure out that there is a looooong way away from where we are today and this feature. There's some value that we can see in us automating the things that need to happen around a Docker registry (push this, pull that, tag this, etc.) - but, because of the ground we would need to cover to make it happen, we're not going to be able to prioritize this for now.

There are things that we currently have roadmapped that would cover the aforementioned gap (today -> auto-Docker) in indirect ways. If this Discussion garners intense interest, that could change things, but, until then, we don't feel comfortable promising anything around this right now.

So, for right now, we'll leave this open for the community to provide input on what they would like to see out of this feature. (If you only want to show that you're interested, please leave a 👍 on the top-level rather than commenting with a +1. Helps those of us on the team sort quickly for what the community is regarding most highly.)

P.S. My input: I love the idea of this and wish we could wave a magic wand and make it happen. 😄

[jakeleventhal]

Thanks for the response. This all make sense

[tiagobento]

As a workaround for users in need of a "hydration" process after restoring cached outputs, I managed to change my pipeline in a way that gives me a "hook" in the cache restoration process.

What I did to my root turbo.json was:

"pipeline:" {
- "build": {
-   "dependsOn": ["^build"],
-   "inputs": ["src/**"],
-   "outputs": ["dist/**"]
- },
+ "build:caching": {
+   "dependsOn": ["^build"],
+   "inputs": ["src/**"],
+   "outputs": ["dist/**"]
+ },
+ "build": {
+   "cache": false,
+   "dependsOn": ["build:caching"]
+ }
}

NOTE: Of course this has massive impact on all packages that were already declaring the build script, since they won't be cached anymore. All build scripts would need to be renamed to build:caching to keep the same behavior.

The build script, in this new configuration, becomes the "after-cached-build" script, which can give you some ability to restore the state of our local docker registry based on a .zip file containing a Docker image, for instance. The majority of my packages are NPM packages, but some are Maven-based. Since Maven artifacts need to be installed to the ~/.m2/repo directory, I see those two use-cases as very similar.

Let me show how I configured my Maven packages. (I removed a lot of stuff for simplicity)

Here's the scripts property on its package.json:

  "scripts": {
    "clean": "mvn clean"
    "build:caching": "mvn package",
    "build": "mvn jar:jar install:install"
  },

This is the turbo.json for this particular Maven package:

{
  "$schema": "https://turbo.build/schema.json",
  "extends": ["//"],
  "pipeline": {
    "build:caching": {
      "outputs": ["**/target/**"]
    }
  }
}

The CPU-bound operation is mvn package, which produces the **/target/** folders with the compiled assets inside. This gets cached. Then the build script will only read from those folders, create a JAR and install the JAR to the ~/.m2/repo directory, where all Maven artifacts are stored locally.

I also have the Docker use-case, but I didn't manage to experiment with it yet.. I'm assuming there's a way to do a very similar process, though.

Hope it helps!