Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for ProgressBars to eta.core.storage #477

Open
brimoor opened this issue Aug 16, 2020 · 0 comments
Open

Add support for ProgressBars to eta.core.storage #477

brimoor opened this issue Aug 16, 2020 · 0 comments
Labels
enhancement Code enhancement

Comments

@brimoor
Copy link
Contributor

brimoor commented Aug 16, 2020

It would be nice to have the option to render a progress bar when uploading/downloading files via the eta.core.storage module, to provide a better experience when transferring large files.

For HTTPSStorageClient, we use requests, so it should be easy enough to support progress bars via the approach outlined in:

eta/eta/core/web.py

Lines 136 to 146 in 03d3101

r = self._get_streaming_response(url, params=params)
etau.ensure_basedir(path)
size_bytes = _get_content_length(r)
size_bits = 8 * size_bytes if size_bytes is not None else None
with etau.ProgressBar(size_bits, use_bits=True) as progress:
with open(path, "wb") as f:
for chunk in r.iter_content(chunk_size=self.chunk_size):
f.write(chunk)
progress.update(8 * len(chunk))

boto3 seems to support callbacks for upload/download functions, which should make this pretty easy for AWSStorageClient:
https://stackoverflow.com/q/41827963

Google Cloud doesn't seem to (yet?) support callbacks, so GoogleCloudStorageClient will take a bit more work:
googleapis/python-storage#27

@brimoor brimoor added the enhancement Code enhancement label Aug 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Code enhancement
Projects
None yet
Development

No branches or pull requests

1 participant