Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ReadSeeker for uploads to s3 #114

Open
mskonovalov opened this issue Sep 5, 2022 · 5 comments
Open

Add support for ReadSeeker for uploads to s3 #114

mskonovalov opened this issue Sep 5, 2022 · 5 comments

Comments

@mskonovalov
Copy link

Unfortunately s3 sdk uses ReadSeeker for uploads.
So in this case it is tricky to wrap a file that you trying to upload: curremt API only allows ReadCloser but not ReadSeeker.

@vbauerster
Copy link
Owner

I never used s3 sdk and probably will not be. IMHO it's horrible approach to use ReadSeeker for uploads.
Nevertheless you can wrap ReadCloser returned by (*Bar).ProxyReader and implement whatever you need.

@mskonovalov
Copy link
Author

@vbauerster I suspect they use ReadSeeker to be able to re-upload specific parts of the file.
That's exactly what I do atm:

ReadSeeker{
	file,
	p.ProxyReader(file),
}

but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%

@vbauerster
Copy link
Owner

but I suspect this will show wrong results in case you try to re-upload the specific part of the file, it easily can get higher 100%

That's correct. There is SetCurrent which may help to keep bar in correct state after seek has been called. If you come up with working implementation, PR is welcome. I'm not able to test against s3 api.

@mskonovalov
Copy link
Author

So I started to test and unfortunately it doesn't work completely:
AWS SDK reads body multiple times to calculate different hashes and so on :(
Because the reader is seekable they can do it without issues.
Don't have ideas yet how to solve it

@mskonovalov
Copy link
Author

mskonovalov commented Sep 14, 2022

Ok, I was able to come up with some weird naive solution (although it is not working properly in some circumstances):


type ProxyReadSeeker struct {
	F           io.ReadSeekCloser
	ProxyReader io.ReadCloser
	Bar         *mpb.Bar
}

func (rs *ProxyReadSeeker) Read(p []byte) (n int, err error) {
	return rs.ProxyReader.Read(p)
}

func (rs *ProxyReadSeeker) Close() error {
	return rs.F.Close()
}

func (rs *ProxyReadSeeker) Seek(offset int64, whence int) (int64, error) {
	if whence == io.SeekStart {
		rs.Bar.SetCurrent(offset)
	}
	return rs.F.Seek(offset, whence)
}

Not sure if it is useful for anyone else, but it seems to work. Although you cannot rely on automatic completion and have to disable it and complete the bar manually.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants