Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rewrite Companion providers to use streams to allow simultaneous upload/download without saving to disk #3159

Merged
merged 51 commits into from Nov 1, 2021

Commits on Jul 26, 2021

  1. rewrite to async/await

    mifi committed Jul 26, 2021
    Configuration menu
    Copy the full SHA
    27c64c7 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ce4e8f3 View commit details
    Browse the repository at this point in the history

Commits on Jul 29, 2021

  1. Update packages/@uppy/companion/src/server/controllers/url.js

    Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
    mifi and aduh95 committed Jul 29, 2021
    Configuration menu
    Copy the full SHA
    b9d2091 View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2021

  1. Configuration menu
    Copy the full SHA
    88b7c51 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2021

  1. Change HEAD to GET in getURLMeta

    and abort request immediately upon response headers received
    #3034 (comment)
    mifi committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    0b6c488 View commit details
    Browse the repository at this point in the history
  2. fix lint

    mifi committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    77c4f25 View commit details
    Browse the repository at this point in the history
  3. fix lint

    mifi committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    4226c95 View commit details
    Browse the repository at this point in the history
  4. cut off length of file names

    or else we get
    "MetadataTooLarge: Your metadata headers exceed the maximum allowed metadata size" in tus / S3
    mifi committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    cf494e5 View commit details
    Browse the repository at this point in the history
  5. try to fix flaky test

    mifi committed Aug 30, 2021
    Configuration menu
    Copy the full SHA
    4aabeea View commit details
    Browse the repository at this point in the history

Commits on Aug 31, 2021

  1. Configuration menu
    Copy the full SHA
    11429a7 View commit details
    Browse the repository at this point in the history
  2. fix lint by reordering code

    mifi committed Aug 31, 2021
    Configuration menu
    Copy the full SHA
    0ba8349 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    55370c5 View commit details
    Browse the repository at this point in the history

Commits on Sep 2, 2021

  1. Rewrite Uploader to use fs-capacitor #3098

    This allows for upload to start almost immediately without having to first download the file.
    And it allows for uploading bigger files, because transloadit assembly will not timeout,
    as it will get upload progress events all the time.
    No longer need for illusive progress.
    Also fix eslint warnings and simplify logic
    
    Still TODO: TUS pause/resume has a bug:
    tus/tus-js-client#275
    mifi committed Sep 2, 2021
    Configuration menu
    Copy the full SHA
    f406288 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    7a08af4 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2d6efdf View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    1a77274 View commit details
    Browse the repository at this point in the history
  5. fix also cancel for xhr

    mifi committed Sep 2, 2021
    Configuration menu
    Copy the full SHA
    1b763be View commit details
    Browse the repository at this point in the history

Commits on Sep 3, 2021

  1. Rewrite providers to use streams

    This removes the need for disk space as data will be buffered in memory and backpressure will be respected
    #3098 (comment)
    All providers "download" methods will now return a { stream } which can be consumed by uploader.
    
    Also:
    - Remove capacitor (no longer needed)
    - Change Provider/SearchProvider API to async (Breaking change for custom companion providers)
    - Fix the case with unknown length streams (zoom / google drive). Need to be downloaded first
    - rewrite controllers deauth-callback, thumbnail, list, logout to async
    - getURLMeta: make sure size is never NaN (NaN gets converted to null in JSON.stringify when sent to client but not when used in backend)
    - fix purest mock (it wasn't returning statusCode on('response'))
    - add missing http mock for "request" for THUMBNAIL_URL and http://url.myendpoint.com/file (these request errors were never caught by tests previously)
    - "upload functions with tus protocol" test: move filename checking to new test where size is null. Fix broken expects
    - fix some lint
    mifi committed Sep 3, 2021
    Configuration menu
    Copy the full SHA
    cdb6239 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    2f7fc2d View commit details
    Browse the repository at this point in the history

Commits on Sep 7, 2021

  1. Implement streamingUpload flag

    COMPANION_STREAMING_UPLOAD
    Default to false due to backward compatibility
    If set to true, will start to upload files at the same time as dowlnoading them, by piping the streams
    
    - Also implement progress for downloading too
    - and fix progress duplication logic
    - fix test that assumed file was fully downloaded after first progress event
    mifi committed Sep 7, 2021
    Configuration menu
    Copy the full SHA
    50cd8c3 View commit details
    Browse the repository at this point in the history
  2. rearrange validation logic

    mifi committed Sep 7, 2021
    Configuration menu
    Copy the full SHA
    bd255aa View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    e756b0a View commit details
    Browse the repository at this point in the history
  4. implement maxFileSize option in companion

    for both unknown length and known length downloads
    mifi committed Sep 7, 2021
    Configuration menu
    Copy the full SHA
    9113472 View commit details
    Browse the repository at this point in the history

Commits on Sep 8, 2021

  1. fix bug

    mifi committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    919ec18 View commit details
    Browse the repository at this point in the history
  2. fix memory leak when non 200 status

    streams were being kept
    mifi committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    eecdba9 View commit details
    Browse the repository at this point in the history
  3. fix lint

    mifi committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    bc62a69 View commit details
    Browse the repository at this point in the history
  4. Add backward-compatibility for companion providers

    Implement a new static field "version" on providers, which when not set to 2,
    will cause a compatibility layer to be added for supporting old callback style provider api
    
    also fix some eslint and rename some vars
    mifi committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    a16f34c View commit details
    Browse the repository at this point in the history
  5. document new provider API

    mifi committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    313d75b View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    65ac8c7 View commit details
    Browse the repository at this point in the history
  7. try to fix build issue

    mifi committed Sep 8, 2021
    Configuration menu
    Copy the full SHA
    c8ff49c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    735bde1 View commit details
    Browse the repository at this point in the history
  9. Configuration menu
    Copy the full SHA
    812a44f View commit details
    Browse the repository at this point in the history
  10. Configuration menu
    Copy the full SHA
    a1bbb62 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2021

  1. fix lint

    mifi committed Sep 9, 2021
    Configuration menu
    Copy the full SHA
    6a6b697 View commit details
    Browse the repository at this point in the history

Commits on Oct 23, 2021

  1. make methods private

    mifi committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    42c2b3d View commit details
    Browse the repository at this point in the history
  2. Merge branch 'main' into companion-upload-while-downloading

    # Conflicts:
    #	package-lock.json
    #	packages/@uppy/companion/src/server/Uploader.js
    #	packages/@uppy/companion/src/server/controllers/url.js
    #	packages/@uppy/companion/src/server/helpers/request.js
    #	packages/@uppy/companion/src/server/provider/unsplash/index.js
    #	packages/@uppy/companion/test/__tests__/uploader.js
    #	website/src/docs/companion.md
    mifi committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    84385bc View commit details
    Browse the repository at this point in the history
  3. re-add unsplash download_location request

    got lost in merge
    mifi committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    e322b23 View commit details
    Browse the repository at this point in the history
  4. add try/catch

    as suggested #3159 (comment)
    mifi committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    4783593 View commit details
    Browse the repository at this point in the history
  5. Only set default chunkSize if needed

    for being more compliant with previous behavior when streamingUpload = false
    mifi committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    670ca8f View commit details
    Browse the repository at this point in the history
  6. Improve flaky test

    Trying to fix this error:
    
    FAIL packages/@uppy/utils/src/delay.test.js
      ● delay › should reject when signal is aborted
    
        expect(received).toBeLessThan(expected)
    
        Expected: < 70
        Received:   107
    
          32 |     const time = Date.now() - start
          33 |     expect(time).toBeGreaterThanOrEqual(30)
        > 34 |     expect(time).toBeLessThan(70)
             |                  ^
          35 |   })
          36 | })
          37 |
    
          at Object.<anonymous> (packages/@uppy/utils/src/delay.test.js:34:18)
    
    https://github.com/transloadit/uppy/runs/3984613454?check_suite_focus=true
    mifi committed Oct 23, 2021
    Configuration menu
    Copy the full SHA
    1e28ef2 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2021

  1. Apply suggestions from code review

    Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
    mifi and aduh95 committed Oct 24, 2021
    Configuration menu
    Copy the full SHA
    d0ff2cc View commit details
    Browse the repository at this point in the history

Commits on Oct 27, 2021

  1. fix review feedback & lint

    mifi committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    9a6af79 View commit details
    Browse the repository at this point in the history
  2. Apply suggestions from code review

    Co-authored-by: Merlijn Vos <merlijn@soverin.net>
    mifi and Murderlon committed Oct 27, 2021
    Configuration menu
    Copy the full SHA
    4eff356 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2021

  1. Merge commit 'e30601df8a89c1a83a6fdb2eb1c8e0eea5cd42cf' into companio…

    …n-upload-while-downloading
    mifi committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    0f148b5 View commit details
    Browse the repository at this point in the history
  2. remove unneeded ts-ignore

    mifi committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    ecb86da View commit details
    Browse the repository at this point in the history
  3. Update packages/@uppy/companion/src/server/controllers/url.js

    Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
    mifi and aduh95 committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    b775d4a View commit details
    Browse the repository at this point in the history
  4. Update packages/@uppy/companion/src/server/Uploader.js

    Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
    mifi and aduh95 committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    259e493 View commit details
    Browse the repository at this point in the history
  5. reduce nesting

    mifi committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    21b9d64 View commit details
    Browse the repository at this point in the history
  6. fix lint

    mifi committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    2338d2f View commit details
    Browse the repository at this point in the history
  7. optimize promisify

    mifi committed Oct 28, 2021
    Configuration menu
    Copy the full SHA
    55a754a View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2021

  1. Update packages/@uppy/companion/test/__tests__/uploader.js

    Co-authored-by: Antoine du Hamel <duhamelantoine1995@gmail.com>
    mifi and aduh95 committed Oct 29, 2021
    Configuration menu
    Copy the full SHA
    291d8e3 View commit details
    Browse the repository at this point in the history