Make AWS S3 expiryWindow configurable to avoid S3 400 Bad Request

[shuaichang]

We got 400 Bad request returned by S3 complaining token expired when accessing S3 with the presigned URL due to S3 token expired due to ec2RoleProvider default expiryWindow to be 5minute.

So the secret token used to presign URL will not be refreshed until 5 minutes prior to expiration. In the worst case the client pull layers have only 5 minutes to complete, this could be a problem for larger layers or slower network.

https://github.com/aws/aws-sdk-go/blob/1371ed99dade3fe52505d9bdcc945f7adecf9810/aws/defaults/defaults.go#L250

Is it possible to consider taking an option for AWS to configure the expiryWindow?

If the idea sounds reasonable, we are happy to make a PR for this

[milosgajdos]

Can you please confirm what version of registry are you running?

[shuaichang]

Thanks for the quick reply, the version is https://github.com/distribution/distribution/tree/v2.7.1

[milosgajdos]

Thanks for the quick reply, the version is https://github.com/distribution/distribution/tree/v2.7.1

Ooof, that's what I suspected. That's an ancient registry release. There have been a few news ones over the past year or two but I would not expect those to fix many issues.

I'd encourage you to take https://hub.docker.com/r/distribution/distribution for a spin. Specifically, the edge tag which gets updated regularly.

We're actively working on a new release that should be due this year.

[shuaichang]

Thanks for the pointer! Looking at the distribution master code, though, it seems the problem still exists unless we have a way to pass in ExpiryWindow for AWS S3 storage driver, is my understanding correct or there some fixes could resolve this token expiration issue?

[milosgajdos]

Thanks for the pointer! Looking at the distribution master code, though, it seems the problem still exists unless we have a way to pass in ExpiryWindow for AWS S3 storage driver, is my understanding correct or there some fixes that could resolve this token expiration issue?

Quite likely. I've never experienced it and would have to double check, but I thought it was worth pointing out that the version you are running is old and likely has a big pile of security issues (among a bunch of bugs that have been fixed in higher versions)

Do you have an example how I could repro this? Is it just the case of a large layer being pulled?

[vl-kp]

i think it is same for 3.0.0-alpha.1

[milosgajdos]

It seems we are hardcoding the expiry on the presigned URL to 20 minutes:

distribution/registry/storage/driver/s3-aws/s3.go

Line 1046 in 38beeee

expiresIn := 20 * time.Minute

We can make this configurable, indeed. It'd be useful on slow networks when pulling large layers as you suggested. Would you be willing to open a PR @shuaichang