Uploading from Google Drive hangs after a while

[mifi]

A person tried to upload a 100GB file many times, but observed that it hangs at 19% (presumably before 40 minutes) with no error in the UI and no error in the companion logs.

I tried to upload a 100GB file from my own google drive, and it seems to progress fine, although I did not try to finish it due to the fact that it will take hours/days to test.

Because companion first downloads the whole file before uploading, but it emits an "Illusive Progress", so if it hung at 19% (less than 50%), it means that it hung while downloading the file from drive. So either something went wrong during downloading on the server (companion), or the client got disconnected from the server after some time. But it's hard to say because no error logged.

I'm thinking it could be due to:

• a firewall cutting the client's connection after a while
• a bug in the companion code or dependencies

I tried to reproduce a server connection cut in companion while uploading and I tried running abort() on the requestStream:

uppy/packages/@uppy/companion/src/server/provider/drive/index.js

Line 235 in d613b84

requestStream

...and when doing that I see that companion stops printing the uploader.illusive.progress events, but it does not emit any error and the UI still shows as uploading.

When I call requestStream.emit('error', new Error('test')), I see it successfully logging the error in companion and also reporting an error in the UI, so it seems to successfully propagate server errors to the UI. This leads me to believe that there is something wrong with request or purest (uses request). request is already deprecated so it is no longer supported on newer versions of Node, so it should be replaced. But I see that purest is very tightly integrated into the codebase in all the providers and also the supporting code.

Also if I'm uploading and then cut the browser's connection with companion, while uploading, it also doesn't report any error. I don't know if that's a feature, but if contact with the server is lost, it could indeed produce the issue that the person had where it gets stuck at 19%.

So I cannot really find the cause of this issue. And rewriting companion to use something else than request a big feat and needs a lot of complicated testing because there are so many providers. We could rewrite just the google drive provider, but it's no guarantee it will fix it.

If someone with a multi-gigabit internet connection could test uploading a 100gb file from google drive, I can share the file with you.

[Murderlon]

Also if I'm uploading and then cut the browser's connection with companion, while uploading, it also doesn't report any error. I don't know if that's a feature, but if contact with the server is lost, it could indeed produce the issue that the person had where it gets stuck at 19%.

Related to #1658?

[mifi]

Looks like it could be, yes

[mifi]

I think I have identified the reason why uploading to transloadit/tus causes the assembly to timeout after 20/40min. As mention earlier, companion will currently completely download the file from google drive (or any other provider) before even starting the upload to TUS. Because downloading a 100gb file can take longer than 20/40 minutes, TUS doesn't get any upload progress events and neither does the assembly because the upload hasn't yet started. Then the assembly will timeout because it doesn't have any data. This assembly timeout might be what causes the progress bar to just stop in the UI without any feedback.

Workarounds that I can think of are:

1. Rewrite companion to download/upload at the same time (e.g. support Node.js streams). Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk #3159
   Issues with this are:

• Some of the upload destinations (s3 uploader or multipart) may not support streams and need a file to upload (can still save as a file for those). (they all support streams)
• We need to support backpressure in our implementation. The current provider API does not support this kind of mechanism (simple onData callback):

  uppy/packages/@uppy/companion/src/server/provider/drive/index.js

  Line 240 in 8e4fae9

  onData(this._error(null, { ...resp, body: jsonResp }))

• So we would need to rewrite the Provider API to a stream based API as well as rewrite and test the 8 different providers and properly handle backpressure.
• Alternatively, maybe rewrite onData to return a promise. Then the sender could await the promise before calling it again. The receiver would resolve the promise when the stream is ready for more data.
• Possibly alternatively just use a dumb passthrough stream with no backpressure support. Then if data is read faster than it can be written (or upload hangs), incoming data will be buffered in memory in Node.js and can cause companion to buffer 100gb or more in memory.
• Possibly implement our own custom passthrough stream implementation that will intelligently buffer the data it receives in a file. Then when the reading end tries to read, it will read from the file at the desired position. If the file has not yet been written up until the requested read position, wait until the file's size is big enough and then read from it (I will try to find an existing node module that does this already).

2. Periodically send "fake" progress events (e.g. 0 bytes) or some kind of keep-alive to the assembly from the frontend (uppy). However I couldn't find any API for this.

Updating companion to use streams gives more advantages like speed and not needing to store huge files on disk, but it requires a bigger rewrite.

[mifi]

The underlying issue causing progress events to not be emitted is now fixed in #3159, so if running companion with the option streamingUpload: true or env COMPANION_STREAMING_UPLOAD=true it should no longer be a problem.