Ideas to improve AppImage

[TheAssassin]

With over 200 MiB, the AppImage is quite large. In this issue, ideas shall be collected how this size can be reduced.

• Remove unused Qt libraries
  • Currently, all Qt plugins are deployed, but I am quite sure not all of them are even used. Removing the unused ones will save quite some space.
• Better compression
  • Try xz vs. gzip compression of AppImage's squashfs image (maybe we should even try zstd, which has recently been added, but isn't supported in appimagetool yet)
  • use UPX to compress libraries -- might not work well with the squashfs compression, so we need to compare all combinations

By using upx on the Qt libraries and using the standard gzip compression, I was able to reduce the size to ~150 MiB already.

We need to compare the startup times when starting to compress files in the AppImage or changing the AppImage's compression to the big one, as they will increase. As disk size is cheap and AppImageUpdate allows for delta updating AppImages, ~100 MiB would still be okay, given there's a whole Python distribution in the AppImage.

[TheLastProject]

For what it's worth, the macOS .app (after extracting), is around 230MB. The .dmg just compresses really well. I definitely agree that this should be improved, but it's already good to have an AppImage in the first place.

[TheAssassin]

There's some tarballs and Windows exe files in the AppDir. They can safely be removed.

[TheAssassin]

I think for every Python file we have a .pyc for, we can delete the .py file, which should save some more space. Perhaps we can compile the files to .pyo files, they should run a bit faster even.

[TheAssassin]

Does Pext use Qt's web engine core plugin? It's among the largest plugin files.

[TheLastProject]

I don't believe it does, no. And sure, those make sense.

[TheAssassin]

lib2to3 and a few other Python libraries are surely not used, and may be removed. lib2to3 is especially annoying, as it causes the python -m compileall call to yield compilation errors, so by removing it we could kill two birds with one stone.

[TheLastProject]

Looking at both the Windows and macOS build, which are both not really optimized either, ~120MB in compressed form should be something reasonably reachable.

[TheAssassin]

That'd mean to cut the current size by half... I'd be curious to see what can be removed safely. Quite some Qt5 libraries could be removed for instance, which could save a few MB. I'm sure there's more to remove.

[looooo]

• conda installs many static libraries. I guess all of them can be deleted. rm lib/*.a
• I guess also header-files can be deleted rm include
• create environment without conda

[TheLastProject]

Looking at the releases, somewhere between 0.18 and 0.19, the AppImage went from 253MB to 377MB. However, there is no change on Pext's side to explain this: v0.18...v0.19.

Something third party must've changed between Aug 22 and Sep 5. Seeing how there is no size change for the macOS or Windows packages, I worry this may have changed on the AppImage side, as that is the only obvious difference between the Linux and other builds.

[TheAssassin]

On the AppImage side, only the runtime size could've increased, but that hasn't happened. I suspect that miniconda's environment grew in size.

By the way, you may want to check out https://github.com/TheAssassin/linuxdeploy-plugin-conda#example. If Pext could be "packaged" as a conda package, AppImage creation could be broken down to a few lines of code using our new, unified AppDir creation tool linuxdeploy.
All the optimizations suggested in this thread are subject to be implemented in that plugin by the way. Thanks for your suggestions!

[TheLastProject]

Packaging as a conda package may be something to look into, but at the same time, it probably would just cause me extra maintenance work because it'd create more difference between the Linux, macOS and Windows build processes. The "remove bloat part" or that plugin, however, is definitely helpful.

Miniconda's environment having grown 120MB in size, but only for Linux, seems a bit dubious to me. I'll try to compare the two AppImages when I get home to a Linux system to see where the growth actually is.

[TheAssassin]

It would've been more dubious if appimagetool had generated that amount of data somehow. That's the only AppImage tool that is used, and the only thing that it ships that will be contained in the AppImage is the runtime. Since the tool still is tiny (just a few MB), that can't have added that bloat, unless it copies the runtime like 100 times.

[TheLastProject]

It was indeed miniconda. Pext's AppImage is now WAY smaller, which is awesome! Will need to look into what Qt modules I actually need :)

[TheAssassin]

Pext's AppImage is now WAY smaller

How so? Did you take any action? If yes, we should implement that in linuxdeploy-plugin-conda.

By the way, linuxdeploy-plugin-conda itself doesn't require you to provide conda packages. Its basic use case is to create a miniconda environment in an AppDir in a sort-of-standardized way. It could replace parts of the AppImage build process in this project. Shall I send a PR for demonstration?

[TheLastProject]

Just the cleanups you also do in linuxdeploy-plugin-conda: 4383ac9.

Hmm, well, following upstream AppImage more does make sense I suppose. The macOS and Windows builds are far from a one-on-one copy anyway, so maybe I can also improve those later by looking at the new linuxdeploy techniques. A PR is welcome :)