Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

storage: upload_from_filename() should use streams (upload() does) #426

Open
awoimbee opened this issue Feb 9, 2022 · 1 comment
Open

Comments

@awoimbee
Copy link

awoimbee commented Feb 9, 2022

This is an issue when uploading large files (it hogs ram until it or the system crashes).

Seems easy to fix: simply remove this line

But then another issue arises, Storage.upload() accepts a synchronous fileobject, but not at async one:

async with aiofiles.open(f"my_file", "rb") as f:
    await gcs_clt.upload(BUCKET_NAME, remote_path, f, timeout=None)
# TypeError: unsupported upload type: "<class 'aiofiles.threadpool.binary.AsyncBufferedReader'>"

with open(f"my_file", "rb") as f:
    await gcs_clt.upload(BUCKET_NAME, remote_path, f, timeout=None)
# All good !
@TheKevJames
Copy link
Member

Unfortunately, this was done intentionally to avoid this bug. We would definitely be interested in a solution which uses streams without triggering that issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

No branches or pull requests

2 participants