Proposal: Buildkit-based remote cache

[vladaionescu]

Buildkit has functionality to store cache in a docker registry. Initial testing showed that it's not just as easy as turning the setting on. Need to explore why that is and address it.

This would make CI integration even better as the system would be much more performant with a cache designed for the purpose.

[vladaionescu]

It seems that the underlying buildkit issue has been fixed and this can be attempted again now.

[TriplEight]

Implementing this you can kick start the selling of the new docker subscription, that one with limited pulls xD
If seriously, can be beneficial with another docker registry.

[vladaionescu]

Some numbers from remote cache experimentation (tested on +for-linux target in Earthly):

Remote cache turned off - baseline metrics:
  First build (cold local): 2m17s
  No change (warm local): 8s
  One line change (warm local): 15s

CI (remote cache is read+write, cold local cache):
  First build (cold upload): 14m01s (upload itself takes 11m52s)
  No change (cold download, warm upload): 2m11s
  One line change (cold download, warm upload): 3m48s

Developer (remote cache is read-only, local cache varies):
  No change, but no local cache (cold download, cold local): 47s
  One line change, no local cache (cold download, cold local): 1m58s
  No change, full local cache (warm download, warm local): 13s
  One line change, some local cache (warm download, warm local): 19s

Notes:

• For CI - it doesn't seem to improve performance for this build
• For developer - note that numbers aren't actually improved. If working with a cold local cache, the system will constantly download more and more pieces on each build, leading to total time increased than it would be if developer would simply run without remote cache (no change + one line change total is increased in remote cache case).
• This is only one build, and only one particular computer (my pretty fast workstation). Perhaps it's more relevant in underpowered setups or compute-intensive builds.

The strategy used was one cache repo for the entire build. Wonder if it's possible for each target to have its own cache. However, it's hard to imagine that that would increase performance.

[vladaionescu]

This has been implemented and results are pretty good especially for inline caching. Docs changes are here #614 .

This will go out in the next release as an experimental feature.

[vladaionescu]

A known issue: moby/buildkit#1902