proxy_cache_convert_head required for pushing to ECR

[aspettl]

Hi,

with ALLOW_PUSH=true, I had problems this week to push to Elastic Container Registry: HTTP 400 with unexpected end of JSON input: "". Finally I found out that Nginx converts HEAD requests to GET by default. Disabling proxy_cache_convert_head fixes the issue for me.

Like the documentation at http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_convert_head says, I also extended the proxy_cache_key to include the request method - however, I'm not sure this is really needed (depends on whether the same address is used with both HEAD and GET - I don't know if this is impossible with the registry protocol or not). Of course, changing the key basically invalidates the whole current cache.

Did someone else experience this, is this ECR-specific? Or do you know whether the cache key could stay as it is without risk of returning wrong responses?
I do not really feel confident of creating a PR, as I'm not sure if changing the request method is really required or not - and that's a breaking change.

Thanks!

[rpardini]

Hi, sorry for the delay in getting back to you.
Disabling HEAD/GET conversion has come up in other issues (I think because of DockerHub rate-limiting, we waste a request depending on dockerd/containerd behavior).
Indeed it requires changing the cache key, which will invalidate the whole cache -- but if you're empty/testing, why not.
PRs are welcome, I usually merge a bunch (some very very similar) every few months and do a release. I'm overdue...

[rpardini]

BTW regarding ECR, I know some people are using it against ECR, but I highly doubt any of them had ALLOW_PUSH too. Could you share your use-case? It is a build farm?

[aspettl]

ALLOW_PUSH is quite new so I was happy to see it exists :-)
The use case is local caching on pull to a) avoid rate limiting problems for a whole k8s cluster that shares one external IP and b) reduce external traffic (building in dynamically created k8s build pods, so no caching in the docker daemon). ECR contains the base images and the built images need to go into the same registry, so allowing to push via the proxy is very useful.

I'll try to create a PR in the next few days.

Edit: just saw #59 (review), maybe should have searched in the issues for HEAD and GET sooner...

[speller]

Can anyone share how to authenticate the proxy on ECR?