Rewrite Companion providers to use streams to allow simultaneous uplo…

…ad/download without saving to disk (transloadit#3159)

* rewrite to async/await

* Only fetch size (HEAD) if needed transloadit#3034

* Update packages/@uppy/companion/src/server/controllers/url.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* Change HEAD to GET in getURLMeta

and abort request immediately upon response headers received
transloadit#3034 (comment)

* fix lint

* fix lint

* cut off length of file names

or else we get
"MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3

* try to fix flaky test

* remove iife and cleanup code a bit

* fix lint by reordering code

* rename Uploader to MultipartUploader

* Rewrite Uploader to use fs-capacitor transloadit#3098

This allows for upload to start almost immediately without having to first download the file.
And it allows for uploading bigger files, because transloadit assembly will not timeout,
as it will get upload progress events all the time.
No longer need for illusive progress.
Also fix eslint warnings and simplify logic

Still TODO: TUS pause/resume has a bug:
tus/tus-js-client#275

* add comment in dev Dashboard and pull out variable

* fix a bug where remote xhr upload would ignore progress events in the UI

* fix bug where s3 multipart cancel wasn't working

* fix also cancel for xhr

* Rewrite providers to use streams

This removes the need for disk space as data will be buffered in memory and backpressure will be respected
transloadit#3098 (comment)
All providers "download" methods will now return a { stream } which can be consumed by uploader.

Also:
- Remove capacitor (no longer needed)
- Change Provider/SearchProvider API to async (Breaking change for custom companion providers)
- Fix the case with unknown length streams (zoom / google drive). Need to be downloaded first
- rewrite controllers deauth-callback, thumbnail, list, logout to async
- getURLMeta: make sure size is never NaN (NaN gets converted to null in JSON.stringify when sent to client but not when used in backend)
- fix purest mock (it wasn't returning statusCode on('response'))
- add missing http mock for "request" for THUMBNAIL_URL and http://url.myendpoint.com/file (these request errors were never caught by tests previously)
- "upload functions with tus protocol" test: move filename checking to new test where size is null. Fix broken expects
- fix some lint

* Implement streamingUpload flag

COMPANION_STREAMING_UPLOAD
Default to false due to backward compatibility
If set to true, will start to upload files at the same time as dowlnoading them, by piping the streams

- Also implement progress for downloading too
- and fix progress duplication logic
- fix test that assumed file was fully downloaded after first progress event

* rearrange validation logic

* add COMPANION_STREAMING_UPLOAD to env.test.sh too

* implement maxFileSize option in companion

for both unknown length and known length downloads

* fix bug

* fix memory leak when non 200 status

streams were being kept

* fix lint

* Add backward-compatibility for companion providers

Implement a new static field "version" on providers, which when not set to 2,
will cause a compatibility layer to be added for supporting old callback style provider api

also fix some eslint and rename some vars

* document new provider API

* remove static as it doesn't work on node 10

* try to fix build issue

* degrade to node 14 in github actions

due to hitting this error: nodejs/node#40030
https://github.com/transloadit/uppy/pull/3159/checks?check_run_id=3544858518

* pull out duplicated logic into reusable function

* fix lint

* make methods private

* re-add unsplash download_location request

got lost in merge

* add try/catch

as suggested transloadit#3159 (comment)

* Only set default chunkSize if needed

for being more compliant with previous behavior when streamingUpload = false

* Improve flaky test

Trying to fix this error:

FAIL packages/@uppy/utils/src/delay.test.js
  ● delay › should reject when signal is aborted

    expect(received).toBeLessThan(expected)

    Expected: < 70
    Received:   107

      32 |     const time = Date.now() - start
      33 |     expect(time).toBeGreaterThanOrEqual(30)
    > 34 |     expect(time).toBeLessThan(70)
         |                  ^
      35 |   })
      36 | })
      37 |

      at Object.<anonymous> (packages/@uppy/utils/src/delay.test.js:34:18)

https://github.com/transloadit/uppy/runs/3984613454?check_suite_focus=true

* Apply suggestions from code review

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* fix review feedback & lint

* Apply suggestions from code review

Co-authored-by: Merlijn Vos <merlijn@soverin.net>

* remove unneeded ts-ignore

* Update packages/@uppy/companion/src/server/controllers/url.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* Update packages/@uppy/companion/src/server/Uploader.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

* reduce nesting

* fix lint

* optimize promisify

transloadit#3159 (comment)

* Update packages/@uppy/companion/test/__tests__/uploader.js

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>

Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Co-authored-by: Merlijn Vos <merlijn@soverin.net>

KUBERNETES.md

@@ -25,6 +25,7 @@ data:
   COMPANION_DOMAIN: "YOUR SERVER DOMAIN"
   COMPANION_DOMAINS: "sub1.domain.com,sub2.domain.com,sub3.domain.com"
   COMPANION_PROTOCOL: "YOUR SERVER PROTOCOL"
+  COMPANION_STREAMING_UPLOAD: true
   COMPANION_REDIS_URL: redis://:superSecretPassword@uppy-redis.uppy.svc.cluster.local:6379
   COMPANION_SECRET: "shh!Issa Secret!"
   COMPANION_DROPBOX_KEY: "YOUR DROPBOX KEY"

env.test.sh

@@ -5,6 +5,8 @@ export COMPANION_SELF_ENDPOINT="localhost:3020"
 export COMPANION_HIDE_METRICS="false"
 export COMPANION_HIDE_WELCOME="false"
 
+export COMPANION_STREAMING_UPLOAD="true"
+
 export COMPANION_PROTOCOL="http"
 export COMPANION_DATADIR="./test/output"
 export COMPANION_SECRET="secret"

env_example

@@ -4,6 +4,7 @@ COMPANION_DOMAIN=uppy.xxxx.com
 COMPANION_SELF_ENDPOINT=uppy.xxxx.com
 COMPANION_HIDE_METRICS=false
 COMPANION_HIDE_WELCOME=false
+COMPANION_STREAMING_UPLOAD=true
 
 COMPANION_PROTOCOL=https
 COMPANION_DATADIR=/mnt/uppy-server-data

package.json

@@ -85,6 +85,8 @@
     "@types/uuid": "3.4.7",
     "@types/webpack": "^5.28.0",
     "@types/ws": "6.0.4",
+    "into-stream": "^6.0.0",
+    "nock": "^13.1.3",
     "supertest": "3.4.2",
     "typescript": "~4.3"
   },

src/companion.js

@@ -39,6 +39,7 @@ const defaultOptions = {
   },
   debug: true,
   logClientVersion: true,
+  streamingUpload: false,
 }
 
 // make the errors available publicly for custom providers