Explicit cache does not work with AWS ECR due to wrong manifest mediatype

[mcheshkov]

What went wrong?
When I try to run earthly with --remote-cache pointing to ECR and --push it fails on the very last step after pushing all blobs with Error: error writing manifest blob: failed commit on ref "sha256:823...540": unexpected status from PUT request to https://123.dkr.ecr.eu-central-1.amazonaws.com/v2/build-cache/manifests/explicit-cache-test: 400 Bad Request

To reproduce:

VERSION 0.7

foo:
    FROM docker.io/debian:bookworm-20230904
    RUN echo foo

Login to ECR, create private registry and run something like earthly --remote-cache=123.dkr.ecr.eu-central-1.amazonaws.com/build-cache:foo-explicit-cache --push +foo (use your ECR id, region and repository name).

What should have happened?

Since support for cache manifest is already implemented in ECR (according to aws/containers-roadmap#876, ticket remains opened just for discussion), I expected this to work out-of-box.

Looking closely to said ticket, it seems that ECR would accept only application/vnd.oci.image.manifest.v1+json as a manifest mediatype, while BuildKit (and, transitively, earthly) use application/vnd.oci.image.index.v1+json by default.

I tried building v0.7.23 from source with this patch on builder/solver.go, and ECR happily accepted push.

diff --git a/builder/solver.go b/builder/solver.go
index be22709a..6cba60ec 100644
--- a/builder/solver.go
+++ b/builder/solver.go
@@ -164,6 +164,8 @@ func newCacheImportOpt(ref string) client.CacheOptionsEntry {
 func newCacheExportOpt(ref string, max bool) client.CacheOptionsEntry {
        registryCacheOptAttrs := make(map[string]string)
        registryCacheOptAttrs["ref"] = ref
+       registryCacheOptAttrs["oci-mediatypes"] = "true"
+       registryCacheOptAttrs["image-manifest"] = "true"
        if max {
                registryCacheOptAttrs["mode"] = "max"
        }

I'm not sure that this solution is a proper way to go, maybe other registry implementations would not like it that way.
I did not find a way to work around this without rebuilding earhtly client executable.

Logs from buildkitd and command output were not very helpful, all basically saying the same, 400 Bad Request without firther explanation. Maybe that's on ECR, did not check all the way here.

[mcheshkov]

Well, now documentation says that AWS ECR supports explicit cache
https://docs.earthly.dev/docs/caching/caching-via-registry

I'm still receiving same 400 Bad Request when trying to push cache.

Documentation was changed at #3814, with a reference to https://aws.amazon.com/blogs/containers/announcing-remote-cache-support-in-amazon-ecr-for-buildkit-clients/
But that exact article points that one should use image-manifest=true,oci-mediatypes=true to use BuildKit remote cache.

BuildKit introduced the ability to export remote caches in 2020. ... However, the format used for storing these caches was not an Open Containers Initiative (OCI) type. Amazon ECR is an OCI-compliant registry, which means that pushing this remote cache format to Amazon ECR resulted in a validation failure.

The new context key introduced in Buildkit 0.12 here is image-manifest. Setting this key’s value to true lets you now store an OCI-compatible version of a remote cache in the registry. We also setting oci-mediatypes to true since that’s required in order to use image-manifest.

cc @tontinton @alexcb from documentation PR: did you check that it actually works? And, if so, can you help me to debug this?

[alexcb]

We recently had a PR submitted (and merged) for this: #3869

it hasn't been released yet, but you could give it a try using our pre-release binaries from https://github.com/earthly/earthly-staging/releases/tag/v0.1710273402.818533702

[mcheshkov]

Thanks for quick response!

$ earthly --version
earthly version v0.1710273402.818533702 30c9d546d8fb1f8cbc97ba0a18546916e1570fde linux/amd64; Fedora Linux 39.20240313.0 (Silverblue)

On this version earthly --remote-cache=123.dkr.ecr.eu-central-1.amazonaws.com/build-cache:mcheshkov-explicit-cache --max-remote-cache --push ... still fails, as expected
But earthly --remote-cache=123.dkr.ecr.eu-central-1.amazonaws.com/build-cache:mcheshkov-explicit-cache,image-manifest=true,oci-mediatypes=true --max-remote-cache --push ... succeded!

I'm still checking if it works fine with complex builds, but at least mediatype issue looks resolved. Looking forward to release with fixes

[alexcb]

awesome! thanks for confirming that worked once you added in the extra attributes. We'll likely get a new release out next week.