storage - Truncated upload due to length-calculation prior to utf-8 encoding

[jack1323]

Encoding utf-8 can increase the calculated length of an object. The length is calculated prior to encoding. If special characters are in the string, the length will be incorrect and the result is a truncated upload.

in upload(), this:

        stream = self._preprocess_data(file_data)
        content_length = self._get_stream_len(stream)

happens before this in _upload_multipart():

       raw_body: AnyStr = stream.read()
       if isinstance(raw_body, str):
            bytes_body: bytes = raw_body.encode('utf-8')

[TheKevJames]

Ahh, awesome find! I would be more than happy to accept a PR which solves this.

[AdeelK93]

I'm having this issue as well, what would a fix to this look like?

[AdeelK93]

the fix, if anyone is wondering, is to encode as bytes

[TheKevJames]

@AdeelK93 do you happen to have sample code for how you've done this? I would love to get this patched.

[AdeelK93]

sure, this is basically the fix in my code. for a utf8 instance of my_string, i do the following

async with aiohttp.ClientSession() as session:
    client = Storage(session=session)
    await client.upload(bucket, blob, my_string.encode())

[AdeelK93]

or in other words, dont ever upload a string, only upload bytes

[TheKevJames]

Huh, ok, then I'm a bit confused: we do indeed calculate the content length before encoding... but for multipart uploads we actually recompute that length after we do the bytes encode.

Does anyone have a sample file which I could test with to understand this behaviour? I've just tried a couple str uploads which special characters and all seems to be working just fine.

[AdeelK93]

try an emoji for the simplest example, or for something more robust, try this test file.

for the second file, you'll notice that the original file is 27kb, but the file that gets uploaded to GCS is 22kb.

you're not going to get an error. you're going to get an incomplete upload.

then try again with .encode() - the full 27kb file will be uploaded.