The problem with big files

[szbk]

I want to reach the size of the video files on the device. Low size videos are okay (eg 50mb). But if the file size is too big, it takes too long (eg 4gb). Sometimes it doesn't work. How can I directly access the path of the file I selected? When the file size is large, cache loading is a problem. I am having this problem on android.

[miguelpruivo]

@sbilketay there isn't much to do here, unfortunately. Files are copy cached in chunks which is great to prevent memory issues, however, 4GB is too much for some devices to handle as you need always 4GB available + some space left.

If you have devices with 10/15 GB of left storage, it shouldn't be an issue (other than just taking the time to download it).

[szbk]

I'm not interested in downloading files. I want to get the size of a video that is already in the gallery. Is there a way to do this without waiting?

[miguelpruivo]

@sbilketay right now that isn’t possible. But it’s possible to add as a feature.

[tmthecoder]

I'm facing this same issue, caching the file then loading for larger files proves to be an issue when reading the files on devices, especially lower performance android devices. Is there a way to append an option to get a direct path?

[szbk]

☝️

[miguelpruivo]

@tmthecoder @sbilketay that's not possible since SDK 30. It is only possible on Android native (through URIs which allows you to manipulate files directly without relying on its original path, however Flutter doesn't support native Android URIs and needs absolute paths for files).

You can read more about it here in the first question.

[tmthecoder]

Oh, my bad, that's right I forgot about the Scoped Storage stuff on Android. I'm still wondering if there's a way to speed up the process as caching is one of the longest processes in my file work process.

[lrorpilla]

I just want to write that I critically need a solution to this issue as well. My app is primarily a video player and some files I want to play are 4GB huge and crashes at file selection.

I can't seem to implement something like resuming video playback because that would require the video to be cached in the app's scoped storage.

My temporary solution is to offer separate versions for users below Android 11 and those on or above it, which is a bit of extra work. With legacy storage, videos play instantly. There isn't copying involved. No videos that have to be moved to cache and clearing after use. No videos that are too huge to play. It just worked.

Scoped storage has been incredibly high maintainance. Secure at the cost of mind numbingly slow limitations. I understand this must be hard to deal with but I would really appreciate any word on a solution to this problem.

[miguelpruivo]

@lrorpilla unfortunately, the only possible solution for this, is if Dart allows us to create File instances with plataform URIs, at least for Android. There’s no way around since even on native side you don’t use paths to access the file’s contents.

Other way I can think is if we could stream the data directly from each platform, which is doable with some effort I believe but might not fit your use case as well, since you need a File — that’s the bottleneck.

[isAlmogK]

Following this as we also ran into this issue

[tmthecoder]

Maybe we can add some sort of 'withDirect' parameter when opening the file which will stream the data directly from the platform (only applicable to android?). I'd be open to helping write this if needed

[yullg]

@lrorpilla unfortunately, the only possible solution for this, is if Dart allows us to create File instances with plataform URIs, at least for Android. There’s no way around since even on native side you don’t use paths to access the file’s contents.

Other way I can think is if we could stream the data directly from each platform, which is doable with some effort I believe but might not fit your use case as well, since you need a File — that’s the bottleneck.

If the APP has MANAGE_EXTERNAL_STORAGE permission, is it possible to use the original path directly?

[tony123S]

I get memory leakage issue when select a pdf file with size 40mb

[miguelpruivo]

Maybe we can add some sort of 'withDirect' parameter when opening the file which will stream the data directly from the platform (only applicable to android?). I'd be open to helping write this if needed

What would you return to Dart in this case?

[tmthecoder]

What would you return to Dart in this case?

I was thinking of a callback-style function that we can listen on in dart and add to a Stream with the bytes.

So my thoughts are: Start a BufferedReader in Java --> call Dart callback with the read bytes (of predetermined chunk size) --> add to a Stream<List<int>> that users can listen on

[miguelpruivo]

@tmthecoder well, that’s sort of the same that happens when you enable withReadStream while picking. Have you already tried?

[tmthecoder]

@miguelpruivo The withReadStream parameter helps in terms of convenience, but the stream is created from dart, so the file on Android has already been copied over to a locally accessible one, which is what I was aiming to stop. If we use the DocumentFile API to access the file on the Android end (if a specific parameter is set) we can bypass the copying and offer a direct stream of the file's data

[shilangyu]

Is there no way of disabling this caching and moving the responsibility to package users? Picking large files is unbearably slow. package:image_picker for example returns XFiles and explicitly states in the docs that using the path can cause errors and it is recommended to create a local copy. Is there then a similar alternative available in package:file_picker?

This also makes me wonder, how apps such as instagram are able to flawlessly handle huge local files (video) without any noticeable delay for caching.

[Dampfwalze]

I have this problem too, I am trying to pick files of an external USB device. The whole point of this is that I don't have enough memory on my device. This caching problem defeats the whole purpose of this! As a workaround, I am currently using getDirectoryPath() instead. This gives me the actual paths of the files and this works perfectly (not counting the hassle, that you can't really select the Videos you want). Then I am wondering why it isn't possible to get the path to the actual file with pickFiles() too.

Proposals:
Maybe you could use Symbolic Links to create a link in the cache that points to the actual file! The problem with this may be that FAT file systems (so most SD Cards) don't support them. But since the app cache should be in the internal storage, which, at least what I am aware of, is normally not a FAT file system, this should be no Problem!

Also, you could implement a switch to not cache files and just return an error, if the file exceeds a specified file size. Maybe even dependent on the storage left on the device. This wouldn't really solve the main Problem, but at least, the Application still works if the User tries to cache a file that is too large.

I really hope this can and will be fixed soon!

[miguelpruivo]

Android guidelines don’t allow you to have absolute path access since SDK 30 for files. It’s probably working for you when you use the directory path to access the files, but using the native API to launch the picker and then handling the intent result with the files, don’t allow me to access the absolute paths.

It was possible before, but then refactored due to this limitation. The recommended way to handle the original file is through its URI which is provided in the identifier property but isn’t much useful for a Dart File instance.

I’m not sure if what you’re asking is possible at all at this point, tbh.

[msarkrish]

@miguelpruivo Is there any way to restrict the file size while selecting file?

[miguelpruivo]

@msarkrish not that I'm aware of (at least, in all platforms). But you can always check the size after the user picks the file and then show an error to the user that the file is too big and discard it.

[msarkrish]

@miguelpruivo I can check the file size after user picks the file. The problem with this behavior is If user select the large file(Example: 3GB) then it is not working as expected. So my question is like we pass allowed extension is there any way to pass file size as parameter? If you add this option it will be useful in this scenario.

[miguelpruivo]

@msarkrish that's doable however what would happen in that case? Would those files be automatically discarded and the user could think it was a bug? It's hard to implement a clean spec for that use case.

Other option is throwing an error if an user picks a file bigger than X size, that could probably work, although not the best IMO. I'd rather have the pickers to not allow picking files bigger than X size, but it seems that is just a platform limitation.

[msarkrish]

@miguelpruivo I can understand. But Do you have any other solution or idea for my problem? MY problem is If i select more than 3GB or big file then it is not sending the selected file and if try to pick another file then it will throw exception as FilePicker is already running. How to fix this issue?

[progid]

I have the same problem

[abhi16180]

@szbk @lrorpilla I have tweaked it to prevent unnecessary caching. In Android we cannot get direct paths sometimes especially when file in in downloads folder. This tweaked version doesn't cache every single file. It will retrieve the paths directly most of the time. Files will be cached only if path cannot be retrieved directly.
You can try it by using this repo link in pubspec.yaml file.
Tweaked file-picker

[lrorpilla]

Thanks @abhi16180, I'll try to look into your fork sometime.

Just as an update because I still subscribe to this issue, I use filesystem_picker and supply root paths from external_path as an initial path and just list all the files in a directory and display them all in Flutter UI.

Really my use case just requires the absolute file path to be able to make a file for a video player, so I just use a non-native file picker made with Flutter that lists everything in a directory and allows navigating and changing directories. Really being able to list every file in a directory allows me to have the absolute path I need.

It's unfortunate I couldn't do this with the native picker plugins like this one, but to people who have a similar use case, you could consider what I did, it also allows you to have control over your UI.

[abhi16180]

Thanks @abhi16180, I'll try to look into your fork sometime.

Just as an update because I still subscribe to this issue, I use filesystem_picker and supply root paths from external_path as an initial path and just list all the files in a directory and display them all in Flutter UI.

Really my use case just requires the absolute file path to be able to make a file for a video player, so I just use a non-native file picker made with Flutter that lists everything in a directory and allows navigating and changing directories. Really being able to list every file in a directory allows me to have the absolute path I need.

It's unfortunate I couldn't do this with the native picker plugins like this one, but to people who have a similar use case, you could consider what I did, it also allows you to have control over your UI.

Yeah I will look into filesystem_picker package.
And there is a flutter file explorer application,it lists out all directories and file paths including downloads directory. I haven't checked how exactly it's been implemented. But it's really cool.
👇
FileX

[cnoons-cyq]

Hey @lrorpilla, I've been trying to do something similar to what you've explained here but I haven't dug too deeply to solve my issues yet.

Really my use case just requires the absolute file path to be able to make a file for a video player, so I just use a non-native file picker made with Flutter that lists everything in a directory and allows navigating and changing directories. Really being able to list every file in a directory allows me to have the absolute path I need.

I'll have to take a look into filesystem_picker myself, as I am aiming to display a list of videos too but from external storage. When you use filesystem_picker to list all of your video files in a UI to select for playing, do you then experience caching when you select a file to play or are you playing using the direct path?

[lrorpilla]

@cnoons-cyq
It lets me pick the direct path. I then have access to relative files. My use case for using this is when I am picking a video file and I also want to get an SRT, or if I'm picking an HTML file and I also need access to relative images in a relative directory.

I've forked filesystem_picker for my purposes and it works great with no caching -- you get the absolute path you picked. I use it alongside external_paths to get roots to the internal storage and micro SD card if it exists on a device.

[miguelpruivo]

Is #1258 similar to this?

[abhi16180]

@miguelpruivo

Android 10+ still allows to access real path with "MANAGE_EXTERNAL_STORAGE" permission. I'm not familiar with native Android development, but I would suggest the below approach,

1. Ask permission to access the entire file system, and list out real paths.
2. If the user refuses to grant full permission, just fallback to current implementation.

[miguelpruivo]

@miguelpruivo

Android 10+ still allows to access real path with "MANAGE_EXTERNAL_STORAGE" permission. I'm not familiar with native Android development, but I would suggest the below approach,

1. Ask permission to access the entire file system, and list out real paths.
2. If the user refuses to grant full permission, just fallback to current implementation.

I got your point but I don't intend to let the users access/manipulate the original files directly with this to prevent, somehow, unexpected implementations and bad behavior when the user accidentally edits an original file in any given app. The original approach of the official image_picker was to always cache/copy the files, so I follow the same path as per good practices. The user can always delete the original file later if he wants to. Also, having to support this would make it a requisite for all platforms or else in Android it could pick the original file and in iOS the users would pick a copy. The dev would have to handle this cases.

Having a cache/copy behaves similar to mobile platforms. But I'm open to discuss alternatives.

[tomekit]

How about skipping expensive cache/copy and exposing content URI's on Android which then can be converted to Stream using this package: https://pub.dev/packages/uri_content ?

I've made this work, please find below update.

UPDATE:
I've slightly modified the Android code to disable copy: 35f5c6d and gets the size not from the file, but content URI, so no slow disk operation is required.
Disclaimer: This probably breaks: withData: true setting on Android, in which case empty bytes might be returned.

On the Flutter side user can get:

FilePickerResult? result = await FilePicker.platform.pickFiles(
  allowMultiple: true,
  allowCompression: false, 
  withReadStream: UniversalPlatform.isWeb
);

final uriContent = UriContent(); //  https://pub.dev/packages/uri_content 
for (final file in result.files) {
   final uri = Uri.parse(file.identifier!);
   final stream = uriContent.getContentStream(uri);
   // Then you can pass the stream to and enjoy your app not crashing if big file is used, alternatively if you need to process stream all at once, then use: "content = await stream.expand((element) => element).toList();"
}

The only drawback so far that I've experienced is that: getContentStream itself uses Platform code and won't work from within the Isolate / Compute.

[theskyblockman]

flutter/flutter#147037 could solve this issue directly with Flutter's XFile (or a new Flutter package using XFiles). XFile support got added in file_picker so if this is done files won't need to be copied to be accessed. If a contributor could give feedback on if one of the option I proposed is feasible to implement effectively in file_picker. (the second one looks like it is the best one)

[J-avery32]

Are there any PRs for this issue or plans to fix it?

[tomekit]

It seems that there are two possible solutions on the radar.

a) flutter/flutter#147037 is feasible once implemented by Flutter team as mentioned by @theskyblockman comment above.

b) Solution mentioned here: #561 (comment) already works, except it doesn't work from a background isolate until Flutter team resolves this: flutter/flutter#119207 where they already make some progress.