@uppy/companion: increase max limits for remote file list operations #4417

mifi · 2023-04-15T16:45:54Z

...because adding 10k files still takes about 5 minutes (to paginate all files), see discussion here: #4389

advantages:

adding huge folder as much faster
listing files is faster when scrolling (when a lot of files), because less /list API calls

disadvantages:

it takes longer to show the user the first list (when a lot of files) and more data will be used (e.g. mobile)

TODO

All other providers?

arturi · 2023-04-18T15:22:32Z

API limits to pageSize: 100, because we have too many fields: https://stackoverflow.com/questions/42592125/list-request-page-size-being-ignored

If we can reduce the number of fields, maybe we can get more:

const DRIVE_FILE_FIELDS = 'kind,id,imageMediaMetadata,name,mimeType,ownedByMe,permissions(role,emailAddress),size,modifiedTime,iconLink,thumbnailLink,teamDriveId,videoMediaMetadata,shortcutDetails(targetId,targetMimeType),parents'

This reverts commit 70daa70.

| Package | Version | Package | Version | | ------------------------- | ------- | ------------------------- | ------- | | @uppy/angular | 0.5.2 | @uppy/progress-bar | 3.0.2 | | @uppy/audio | 1.1.1 | @uppy/provider-views | 3.3.0 | | @uppy/aws-s3 | 3.1.0 | @uppy/react | 3.1.2 | | @uppy/aws-s3-multipart | 3.2.0 | @uppy/react-native | 0.5.1 | | @uppy/box | 2.1.1 | @uppy/redux-dev-tools | 3.0.2 | | @uppy/companion | 4.5.0 | @uppy/remote-sources | 1.0.3 | | @uppy/companion-client | 3.1.3 | @uppy/screen-capture | 3.1.1 | | @uppy/compressor | 1.0.2 | @uppy/status-bar | 3.1.1 | | @uppy/core | 3.2.0 | @uppy/store-default | 3.0.3 | | @uppy/dashboard | 3.4.0 | @uppy/store-redux | 3.0.3 | | @uppy/drag-drop | 3.0.2 | @uppy/svelte | 3.0.2 | | @uppy/dropbox | 3.1.1 | @uppy/thumbnail-generator | 3.0.3 | | @uppy/facebook | 3.1.1 | @uppy/transloadit | 3.1.3 | | @uppy/file-input | 3.0.2 | @uppy/tus | 3.1.0 | | @uppy/form | 3.0.2 | @uppy/unsplash | 3.2.1 | | @uppy/golden-retriever | 3.0.3 | @uppy/url | 3.3.1 | | @uppy/google-drive | 3.1.1 | @uppy/utils | 5.3.0 | | @uppy/image-editor | 2.1.2 | @uppy/vue | 1.0.2 | | @uppy/informer | 3.0.2 | @uppy/webcam | 3.3.1 | | @uppy/instagram | 3.1.1 | @uppy/xhr-upload | 3.2.0 | | @uppy/locales | 3.2.0 | @uppy/zoom | 2.1.1 | | @uppy/onedrive | 3.1.1 | uppy | 3.8.0 | - @uppy/companion: increase max limits for remote file list operations (Mikael Finstad / #4417) - @uppy/xhr-upload: fix type in README.md (Top Master / #4416) - @uppy/core: improve performance of validating & uploading files (Mikael Finstad / #4402) - @uppy/provider-views: Concurrent file listing (Mikael Finstad / #4401) - @uppy/core,@uppy/locales,@uppy/provider-views: User feedback adding recursive folders take 2 (Mikael Finstad / #4399) - @uppy/dashboard: Single File Mode: fix layout and make optional (Artur Paikin / #4374) - @uppy/informer: add a check in `TransitionGroup` when component is null (Juan Belej / #4410) - meta: Fix logos in all the readmes (Artur Paikin / #4407) - meta: fix logo in readme (Kid / #4403)

…ransloadit#4417) * increase max limits for list operations * fix broken test * pageSize won't work until we get rid of permissions field * Revert "fix broken test" This reverts commit 70daa70. --------- Co-authored-by: Artur Paikin <artur@arturpaikin.com>

| Package | Version | Package | Version | | ------------------------- | ------- | ------------------------- | ------- | | @uppy/angular | 0.5.2 | @uppy/progress-bar | 3.0.2 | | @uppy/audio | 1.1.1 | @uppy/provider-views | 3.3.0 | | @uppy/aws-s3 | 3.1.0 | @uppy/react | 3.1.2 | | @uppy/aws-s3-multipart | 3.2.0 | @uppy/react-native | 0.5.1 | | @uppy/box | 2.1.1 | @uppy/redux-dev-tools | 3.0.2 | | @uppy/companion | 4.5.0 | @uppy/remote-sources | 1.0.3 | | @uppy/companion-client | 3.1.3 | @uppy/screen-capture | 3.1.1 | | @uppy/compressor | 1.0.2 | @uppy/status-bar | 3.1.1 | | @uppy/core | 3.2.0 | @uppy/store-default | 3.0.3 | | @uppy/dashboard | 3.4.0 | @uppy/store-redux | 3.0.3 | | @uppy/drag-drop | 3.0.2 | @uppy/svelte | 3.0.2 | | @uppy/dropbox | 3.1.1 | @uppy/thumbnail-generator | 3.0.3 | | @uppy/facebook | 3.1.1 | @uppy/transloadit | 3.1.3 | | @uppy/file-input | 3.0.2 | @uppy/tus | 3.1.0 | | @uppy/form | 3.0.2 | @uppy/unsplash | 3.2.1 | | @uppy/golden-retriever | 3.0.3 | @uppy/url | 3.3.1 | | @uppy/google-drive | 3.1.1 | @uppy/utils | 5.3.0 | | @uppy/image-editor | 2.1.2 | @uppy/vue | 1.0.2 | | @uppy/informer | 3.0.2 | @uppy/webcam | 3.3.1 | | @uppy/instagram | 3.1.1 | @uppy/xhr-upload | 3.2.0 | | @uppy/locales | 3.2.0 | @uppy/zoom | 2.1.1 | | @uppy/onedrive | 3.1.1 | uppy | 3.8.0 | - @uppy/companion: increase max limits for remote file list operations (Mikael Finstad / transloadit#4417) - @uppy/xhr-upload: fix type in README.md (Top Master / transloadit#4416) - @uppy/core: improve performance of validating & uploading files (Mikael Finstad / transloadit#4402) - @uppy/provider-views: Concurrent file listing (Mikael Finstad / transloadit#4401) - @uppy/core,@uppy/locales,@uppy/provider-views: User feedback adding recursive folders take 2 (Mikael Finstad / transloadit#4399) - @uppy/dashboard: Single File Mode: fix layout and make optional (Artur Paikin / transloadit#4374) - @uppy/informer: add a check in `TransitionGroup` when component is null (Juan Belej / transloadit#4410) - meta: Fix logos in all the readmes (Artur Paikin / transloadit#4407) - meta: fix logo in readme (Kid / transloadit#4403)

increase max limits for list operations

16cf409

Murderlon changed the title ~~increase max limits for remote file list operations~~ @uppy/companion: increase max limits for remote file list operations Apr 17, 2023

Murderlon approved these changes Apr 17, 2023

View reviewed changes

fix broken test

70daa70

arturi added 2 commits April 18, 2023 17:12

pageSize won't work until we get rid of permissions field

a9a87b5

Revert "fix broken test"

f4d1f4a

This reverts commit 70daa70.

arturi merged commit 80a8275 into main Apr 18, 2023
19 checks passed

arturi deleted the companion-list-max branch April 18, 2023 17:04

github-actions bot mentioned this pull request Apr 18, 2023

Release: uppy@3.8.0 #4418

Merged

H4NG-MAN mentioned this pull request May 12, 2023

[Snyk] Security upgrade hexo from 3.8.0 to 5.0.0 H4NG-MAN/uppy#166

Open

This was referenced Nov 27, 2023

[Snyk] Fix for 14 vulnerabilities H4NG-MAN/uppy#180

Open

[Snyk] Fix for 1 vulnerabilities H4NG-MAN/uppy#186

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

@uppy/companion: increase max limits for remote file list operations #4417

@uppy/companion: increase max limits for remote file list operations #4417

mifi commented Apr 15, 2023 •

edited

arturi commented Apr 18, 2023

@uppy/companion: increase max limits for remote file list operations #4417

@uppy/companion: increase max limits for remote file list operations #4417

Conversation

mifi commented Apr 15, 2023 • edited

TODO

arturi commented Apr 18, 2023

mifi commented Apr 15, 2023 •

edited