Feature request: avoid repeated unpacking of exactly the same app

[bersbersbers]

Is your feature request related to a problem? Please describe.
onefile mode is great for distribution. No zip files, no installer, just copy an exe and run. However, onefile is slow, because every time it runs it extracts the exact same data into almost the same location. In my case, we are speaking 4.5 vs. 1.5 seconds launch times (hand-stopped).

Describe the solution you'd like
I would find it tremendously useful if I could opt-in to a process that works somewhat like this:

• During generation of the onefile bundle, define an identifier for the app and generate a unique ID (maybe a hash of the content) that can be quickly accessed at run-time
• During run-time, check if $TEMP\_MEI_$APP\$UNIQUE exists
  • yes: launch app from that location without extracting further
  • no: extract app into that location and run from there
• On close, remove other/older folders in $TEMP\_MEI_$APP that are not $UNIQUE.

One could name this hybrid mode, or persistent onefile mode, or something that effect.

[rokm]

This comes up every now and then (see for example #7879), but as far as I'm concerned, the answer remains no. The bootloader code is complicated enough as it is.

[bersbersbers]

This comes up every now and then

Maybe that's a sign :) I'll add to the discussion in #7879.

[bwoodsend]

On close, remove other/older folders in $TEMP_MEI_$APP that are not $UNIQUE.

That would mean that if you have two unrelated applications built by PyInstaller, they'd be forever deleting each other's unpacked directories -- possibly even whilst the other application is still running.

At a slightly more persnickety level, there's also the joys of concurrency to deal with -- launch the app twice in quick succession so that the second instance sees the partially unpacked directory, skips its own unpacking and crashes because the extraction isn't finished yet. And what would happen if the application was aborted midway through unpacking -- $TEMP\_MEI_$APP\$UNIQUE would always exist so the next run would always skip the unpack and crash every time it launches.

Also, bear in mind that this all has to be written in C. All this filtering will have to be written from scratch. It has to be platform and locale agnostic, unicode safe, multi-user safe and combed through for anything that can be tricked into modifying or executing the wrong file. The cost of this feature is a lot higher than you seem to realise.

[bersbersbers]

On close, remove other/older folders in $TEMP_MEI_$APP that are not $UNIQUE.

That would mean that if you have two unrelated applications built by PyInstaller, they'd be forever deleting each other's unpacked directories [...]

... if those two unrelated applications share the same $APP identifier, yes: $APP needs to be sufficiently unique. That may still happen with different version of the same app, though.

possibly even whilst the other application is still running

That is impossible on standard Windows, but you are right, it's a risk on other operating/file systems.

At a slightly more persnickety level, there's also the joys of concurrency to deal with -- launch the app twice in quick succession so that the second instance sees the partially unpacked directory, skips its own unpacking and crashes because the extraction isn't finished yet. And what would happen if the application was aborted midway through unpacking -- $TEMP\_MEI_$APP\$UNIQUE would always exist so the next run would always skip the unpack and crash every time it launches.

Good points. The extracting application would have to mark the target folder as "complete", e.g., by placing a file inside the folder, once it is done extracting. The following application would have to poll the existing folder for said file, and if that is not created, (prompt the user to) delete the folder after a timeout.

Also, bear in mind that this all has to be written in C. [...] The cost of this feature is a lot higher than you seem to realise.

This being in C is in fact a point I was not aware of. Again, benefit-cost-ratio is a reason that I can easily accept, and this being C underlines that point.

Is does sound like the way to go is using onedir mode and a good SFX library - from your experience with pyinstaller, can you recommend one by any chance? :)

[bwoodsend]

Is does sound like the way to go is using onedir mode and a good SFX library - from your experience with pyinstaller, can you recommend one by any chance? :)

You'll have to expand the SFX acronym for me. The only meaning I know for it is sound effects.

[bersbersbers]

Sorry! https://en.wikipedia.org/wiki/SFX:

Self-extracting archive, a compressed file with an embedded executable to decompress itself

(I must admit I believe that acronym has lost popularity over the decades.)

[bwoodsend]

I've never actually used one. I'm quite fond of 7zip though so I'd probably go with that if I had to pick without trying.

Honestly, I never understood why people don't like zip files. It's just right click -> extract here and you're done.

[bersbersbers]

Honestly, I never understood why people don't like zip files.

Thanks! I think there's at least two reasons. A historic one is that earlier versions of Windows came without unzipping capabilities. Another one is that if you build on exe on CI, you can download and run it without leaving the browser. Compare https://gitlab.com/gitlab-org/gitlab/-/issues/37384, for instance.

[bwoodsend]

A historic one is that earlier versions of Windows came without unzipping capabilities.

I've just noticed that Windows 7 comes with zip file support. The oldest version of Python that PyInstaller supports is 3.8 and Python hasn't supported anything older than Windows 7 since Python 3.4 so unless you downgrade everything about 4 years worth of releases, you shouldn't be able to support these legacy Windows versions anyway.

[bersbersbers]

Yes, that's why I wrote "historic" :) sometimes, however, preferences seem to take a while to change.