New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk #3159
Commits on Jul 26, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 27c64c7 - Browse repository at this point
Copy the full SHA 27c64c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for ce4e8f3 - Browse repository at this point
Copy the full SHA ce4e8f3View commit details
Commits on Jul 29, 2021
-
Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for b9d2091 - Browse repository at this point
Copy the full SHA b9d2091View commit details
Commits on Aug 29, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 88b7c51 - Browse repository at this point
Copy the full SHA 88b7c51View commit details
Commits on Aug 30, 2021
-
Change HEAD to GET in getURLMeta
and abort request immediately upon response headers received #3034 (comment)
Configuration menu - View commit details
-
Copy full SHA for 0b6c488 - Browse repository at this point
Copy the full SHA 0b6c488View commit details -
Configuration menu - View commit details
-
Copy full SHA for 77c4f25 - Browse repository at this point
Copy the full SHA 77c4f25View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4226c95 - Browse repository at this point
Copy the full SHA 4226c95View commit details -
or else we get "MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3
Configuration menu - View commit details
-
Copy full SHA for cf494e5 - Browse repository at this point
Copy the full SHA cf494e5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4aabeea - Browse repository at this point
Copy the full SHA 4aabeeaView commit details
Commits on Aug 31, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 11429a7 - Browse repository at this point
Copy the full SHA 11429a7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0ba8349 - Browse repository at this point
Copy the full SHA 0ba8349View commit details -
Configuration menu - View commit details
-
Copy full SHA for 55370c5 - Browse repository at this point
Copy the full SHA 55370c5View commit details
Commits on Sep 2, 2021
-
Rewrite Uploader to use fs-capacitor #3098
This allows for upload to start almost immediately without having to first download the file. And it allows for uploading bigger files, because transloadit assembly will not timeout, as it will get upload progress events all the time. No longer need for illusive progress. Also fix eslint warnings and simplify logic Still TODO: TUS pause/resume has a bug: tus/tus-js-client#275
Configuration menu - View commit details
-
Copy full SHA for f406288 - Browse repository at this point
Copy the full SHA f406288View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7a08af4 - Browse repository at this point
Copy the full SHA 7a08af4View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2d6efdf - Browse repository at this point
Copy the full SHA 2d6efdfView commit details -
Configuration menu - View commit details
-
Copy full SHA for 1a77274 - Browse repository at this point
Copy the full SHA 1a77274View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1b763be - Browse repository at this point
Copy the full SHA 1b763beView commit details
Commits on Sep 3, 2021
-
Rewrite providers to use streams
This removes the need for disk space as data will be buffered in memory and backpressure will be respected #3098 (comment) All providers "download" methods will now return a { stream } which can be consumed by uploader. Also: - Remove capacitor (no longer needed) - Change Provider/SearchProvider API to async (Breaking change for custom companion providers) - Fix the case with unknown length streams (zoom / google drive). Need to be downloaded first - rewrite controllers deauth-callback, thumbnail, list, logout to async - getURLMeta: make sure size is never NaN (NaN gets converted to null in JSON.stringify when sent to client but not when used in backend) - fix purest mock (it wasn't returning statusCode on('response')) - add missing http mock for "request" for THUMBNAIL_URL and http://url.myendpoint.com/file (these request errors were never caught by tests previously) - "upload functions with tus protocol" test: move filename checking to new test where size is null. Fix broken expects - fix some lint
Configuration menu - View commit details
-
Copy full SHA for cdb6239 - Browse repository at this point
Copy the full SHA cdb6239View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2f7fc2d - Browse repository at this point
Copy the full SHA 2f7fc2dView commit details
Commits on Sep 7, 2021
-
Implement streamingUpload flag
COMPANION_STREAMING_UPLOAD Default to false due to backward compatibility If set to true, will start to upload files at the same time as dowlnoading them, by piping the streams - Also implement progress for downloading too - and fix progress duplication logic - fix test that assumed file was fully downloaded after first progress event
Configuration menu - View commit details
-
Copy full SHA for 50cd8c3 - Browse repository at this point
Copy the full SHA 50cd8c3View commit details -
Configuration menu - View commit details
-
Copy full SHA for bd255aa - Browse repository at this point
Copy the full SHA bd255aaView commit details -
Configuration menu - View commit details
-
Copy full SHA for e756b0a - Browse repository at this point
Copy the full SHA e756b0aView commit details -
implement maxFileSize option in companion
for both unknown length and known length downloads
Configuration menu - View commit details
-
Copy full SHA for 9113472 - Browse repository at this point
Copy the full SHA 9113472View commit details
Commits on Sep 8, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 919ec18 - Browse repository at this point
Copy the full SHA 919ec18View commit details -
Configuration menu - View commit details
-
Copy full SHA for eecdba9 - Browse repository at this point
Copy the full SHA eecdba9View commit details -
Configuration menu - View commit details
-
Copy full SHA for bc62a69 - Browse repository at this point
Copy the full SHA bc62a69View commit details -
Add backward-compatibility for companion providers
Implement a new static field "version" on providers, which when not set to 2, will cause a compatibility layer to be added for supporting old callback style provider api also fix some eslint and rename some vars
Configuration menu - View commit details
-
Copy full SHA for a16f34c - Browse repository at this point
Copy the full SHA a16f34cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 313d75b - Browse repository at this point
Copy the full SHA 313d75bView commit details -
Configuration menu - View commit details
-
Copy full SHA for 65ac8c7 - Browse repository at this point
Copy the full SHA 65ac8c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for c8ff49c - Browse repository at this point
Copy the full SHA c8ff49cView commit details -
degrade to node 14 in github actions
due to hitting this error: nodejs/node#40030 https://github.com/transloadit/uppy/pull/3159/checks?check_run_id=3544858518
Configuration menu - View commit details
-
Copy full SHA for 735bde1 - Browse repository at this point
Copy the full SHA 735bde1View commit details -
Configuration menu - View commit details
-
Copy full SHA for 812a44f - Browse repository at this point
Copy the full SHA 812a44fView commit details -
Configuration menu - View commit details
-
Copy full SHA for a1bbb62 - Browse repository at this point
Copy the full SHA a1bbb62View commit details
Commits on Sep 9, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 6a6b697 - Browse repository at this point
Copy the full SHA 6a6b697View commit details
Commits on Oct 23, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 42c2b3d - Browse repository at this point
Copy the full SHA 42c2b3dView commit details -
Merge branch 'main' into companion-upload-while-downloading
# Conflicts: # package-lock.json # packages/@uppy/companion/src/server/Uploader.js # packages/@uppy/companion/src/server/controllers/url.js # packages/@uppy/companion/src/server/helpers/request.js # packages/@uppy/companion/src/server/provider/unsplash/index.js # packages/@uppy/companion/test/__tests__/uploader.js # website/src/docs/companion.md
Configuration menu - View commit details
-
Copy full SHA for 84385bc - Browse repository at this point
Copy the full SHA 84385bcView commit details -
Configuration menu - View commit details
-
Copy full SHA for e322b23 - Browse repository at this point
Copy the full SHA e322b23View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4783593 - Browse repository at this point
Copy the full SHA 4783593View commit details -
Only set default chunkSize if needed
for being more compliant with previous behavior when streamingUpload = false
Configuration menu - View commit details
-
Copy full SHA for 670ca8f - Browse repository at this point
Copy the full SHA 670ca8fView commit details -
Trying to fix this error: FAIL packages/@uppy/utils/src/delay.test.js ● delay › should reject when signal is aborted expect(received).toBeLessThan(expected) Expected: < 70 Received: 107 32 | const time = Date.now() - start 33 | expect(time).toBeGreaterThanOrEqual(30) > 34 | expect(time).toBeLessThan(70) | ^ 35 | }) 36 | }) 37 | at Object.<anonymous> (packages/@uppy/utils/src/delay.test.js:34:18) https://github.com/transloadit/uppy/runs/3984613454?check_suite_focus=true
Configuration menu - View commit details
-
Copy full SHA for 1e28ef2 - Browse repository at this point
Copy the full SHA 1e28ef2View commit details
Commits on Oct 24, 2021
-
Apply suggestions from code review
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for d0ff2cc - Browse repository at this point
Copy the full SHA d0ff2ccView commit details
Commits on Oct 27, 2021
-
Configuration menu - View commit details
-
Copy full SHA for 9a6af79 - Browse repository at this point
Copy the full SHA 9a6af79View commit details -
Apply suggestions from code review
Co-authored-by: Merlijn Vos <merlijn@soverin.net>
Configuration menu - View commit details
-
Copy full SHA for 4eff356 - Browse repository at this point
Copy the full SHA 4eff356View commit details
Commits on Oct 28, 2021
-
Merge commit 'e30601df8a89c1a83a6fdb2eb1c8e0eea5cd42cf' into companio…
…n-upload-while-downloading
Configuration menu - View commit details
-
Copy full SHA for 0f148b5 - Browse repository at this point
Copy the full SHA 0f148b5View commit details -
Configuration menu - View commit details
-
Copy full SHA for ecb86da - Browse repository at this point
Copy the full SHA ecb86daView commit details -
Update packages/@uppy/companion/src/server/controllers/url.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for b775d4a - Browse repository at this point
Copy the full SHA b775d4aView commit details -
Update packages/@uppy/companion/src/server/Uploader.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 259e493 - Browse repository at this point
Copy the full SHA 259e493View commit details -
Configuration menu - View commit details
-
Copy full SHA for 21b9d64 - Browse repository at this point
Copy the full SHA 21b9d64View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2338d2f - Browse repository at this point
Copy the full SHA 2338d2fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 55a754a - Browse repository at this point
Copy the full SHA 55a754aView commit details
Commits on Oct 29, 2021
-
Update packages/@uppy/companion/test/__tests__/uploader.js
Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
Configuration menu - View commit details
-
Copy full SHA for 291d8e3 - Browse repository at this point
Copy the full SHA 291d8e3View commit details