Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AsyncGenerator support in storage upload #206

Open
langma opened this issue Sep 17, 2020 · 1 comment
Open

AsyncGenerator support in storage upload #206

langma opened this issue Sep 17, 2020 · 1 comment

Comments

@langma
Copy link

langma commented Sep 17, 2020

It would be great if the upload could consume an AsyncGenerator as the stream for upload body. It seems that in https://github.com/talkiq/gcloud-aio/blob/master/storage/gcloud/aio/storage/storage.py#L250 there is explicit check that the uploaded body (stream) must be of type io.IOBase.

At least aiohttp will accept an AsyncGenerator out of the box, so the change would not be that big, but there might be an issue with requests though. Could this be circumvented with something?

My actual use case is that I'm reading a byte stream from a socket and I need to upload that stream as a file to GCS. The current implementation would require either a tempfile, or BytesIO in memory. It would be nice if the buffering would be handled by the network stack.

@TheKevJames
Copy link
Member

We'd definitely be open to accepting a PR implementing this suggestion! I suspect we'd need to ensure that the requests version of this library strips out the relevant check -- might require a change to the py3to2 code generation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants