Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multipart upload issues with Cloudflare R2 #3873

Open
Vad1mo opened this issue Apr 4, 2023 · 11 comments · May be fixed by #3940 or #4327
Open

Multipart upload issues with Cloudflare R2 #3873

Vad1mo opened this issue Apr 4, 2023 · 11 comments · May be fixed by #3940 or #4327

Comments

@Vad1mo
Copy link

Vad1mo commented Apr 4, 2023

Trying to use Cloudflare R2 as a storage backend.

time="2023-04-04T23:23:34.047425141Z" level=error msg="response completed with error" auth.user.name="harbor_registry_user" err.code=unknown err.detail="InvalidPart: There was a problem with the multipart upload.
	status code: 400, request id: , host id: " err.message="unknown error" go.version=go1.19.4 http.request.host

Small images where multipart isn't applied work fine, bigger images fail with the above error message.

@milosgajdos
Copy link
Member

Seems like this is an issue with Cloudflare not with distribution.

@Vad1mo
Copy link
Author

Vad1mo commented Apr 15, 2023

It is not an issue with R2. It appears that other implemented it compatible with other S3 providers including AWS.

URL encoding the multipart upload ID in the query parameter. This is either caused by R2 having differently structured IDs, or some difference in handling non-urlencoded query parameters.

@milosgajdos
Copy link
Member

These two statements seem contradictory to me:

It is not an issue with R2

This is either caused by R2 having differently structured IDs...

Note: I should clarify that I am not trying to imply that R2 is broken in any way - I'm just saying that distribution does not provide support for R2 storage backend and the proclaimed S3<->R2 compatibility does not seem to be quite there. I admit my saying "Seems like this is an issue with Cloudflare not with distribution" came out the wrong way.

@Vad1mo
Copy link
Author

Vad1mo commented Apr 15, 2023

Got it.
What is important to me is that if I resolve the issue and make the Docker distribution R2 compatible, will the PR be accepted upstream.

@Vad1mo Vad1mo changed the title multipart upload issues with Cloudflare R2 Multipart upload issues with Cloudflare R2 Apr 15, 2023
@milosgajdos
Copy link
Member

Please do go ahead and open a PR, indeed. I can't give you guarantees if the PR will be accepted but I can tell you it'll certainly be reviewed and considered. With the PR at least we'll have something concrete to discuss :-)

@ichernev
Copy link

ichernev commented May 5, 2023

@Vad1mo can you point to the exact line/file where this URL encoding is (not) taking place?

@Vad1mo
Copy link
Author

Vad1mo commented May 11, 2023

@ichernev no idea, I was searching for PRs on GH with same/related issues, I didn't have time to look in into docker distribution. Are you in the same boat?

@ichernev
Copy link

@Vad1mo well, yes. I was planning on using R2 (due to the "compatibility" with S3) but I guess it is still an open issue. I might have some time to debug the root cause of this in the coming weeks. If you figure out what the issue is (even without a patch), I can help with moving this forward.

@Vad1mo
Copy link
Author

Vad1mo commented Jun 1, 2023

@tpoxa is working on a fix from our side

@zhming0
Copy link

zhming0 commented Jan 19, 2024

I incremented the chunksize per this suggestion and it seems to make the problem go away in my tests.

storage:
  s3:
    chunksize: 104857600 # 100MiB

Following the thread, if I am not mistaken, the problem seems to be caused by the combining small part feature? Perhaps, this change impacts the logic of combining small parts?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants