AppImage self-updating

[TheLastProject]

As AppImageUpdate isn't something that's installed commonly, Pext will have to check for updates in the AppImage itself. Probably calling appimageupdatecli from subprocess.

This probably involves modifying _menu_check_updates and _menu_check_updates_actually_check and reverting 1b4b963 and 4120ec7.

I am labelling this a bug because the non-AppImage version does currently check for updates, so it is a regression.

[TheLastProject]

@TheAssassin

[TheLastProject]

I've just reverted 1b4b963 and 4120ec7, at least restoring the old update check mechanism, so changing this to enhancement for it to use AppImage's mechanism.

[TheAssassin]

Do you just need an update check, or do you want full self-updating? In the latter case, it might make sense to ship the AppImageUpdate UI, to provide users with a real user interface.

[TheLastProject]

There is already an update check for stable releases. Full self-updating would be a much better user experience though.

[TheAssassin]

Okay, then let's build a full self-update system.

[TheAssassin]

I decided to write a Python wrapper for libappimageupdate. I always wanted to write some Python bindings, and this seems like the perfect chance for this.

[TheLastProject]

Sounds like a good idea. Please tell me when it's done. Pext would be happy to be an early adopter :)

[TheAssassin]

Almost done. Fixing the last remaining bugs. I will post a link once I'm done. I would like you to implement the update checks and actual updating process, if you don't mind, to have an external person review the library.

[TheLastProject]

Sounds good to me, I'd be happy to :)

[TheAssassin]

Check out https://github.com/TheAssassin/python-appimageupdate.

It's a little buggy, but most of it seems to be libappimageupdate related. We need some unit tests in that library. It's funny that the Python unit tests are bringing up all the errors in libappimageupdate at the moment...

[TheAssassin]

The library is mostly ready, I'm still struggling with the Travis build configuration (and the stop() method that hasn't been implemented in the C++ code properly yet), but other than that the library works fine. I made it as pythonic as possible, the functions throw exceptions on errors instead of returning error states, etc.

Check out https://github.com/AppImage/AppImageUpdate/wiki/Guide-to-libappimageupdate for a guide how to use it. Please make use of the docstrings I wrote in the Python wrapper class, they explain the API in detail.

[TheLastProject]

I'll probably take a look at it this weekend, thank you :)

[TheLastProject]

https://github.com/Pext/Pext/tree/feature/appimageupdate seems to work...ish. There's just a few weird things.

1. After updating, starting the same AppImage will no longer cause "APPIMAGE" to be in the list of environment variables, so Pext no longer understands it is an appimage and can no longer use appimageupdate.
2. This increases the file size by 30MB. This must be possible to make less ridiculously huge.
3. The appimageupdatetool name on https://github.com/AppImage/AppImageUpdate/releases keeps changing, breaking the build script often.

@TheAssassin Any clue on this? Especially 1 and 3 are rather huge issues here.

[TheAssassin]

We can hack around 1 easily by calling setenv("APPIMAGE", pathToAppImage.c_str()) in the code. It's still weird the updated AppImage's runtime doesn't set this variable. I have no explanation for this.

About 2: Please open an issue in the repo, this is certainly a bug.

Regarding 3, why do you need appimageupdatetool at all? Actually, the Python API should expose all the functionality.
We could think about using the same filename for the releases, though. Feel free to open an issue there.

[TheLastProject]

Regarding 1: I'm not sure if such a hack is a good idea, this seems like something that needs to be looked at in AppImageUpdate itself.

Regarding 2: I'm not sure where and how? AppImageUpdate is simply 20MB according to https://github.com/AppImage/AppImageUpdate/releases. I'm not sure how it is a bug? It just is rather big.

Regarding 3: We discussed this before, we needed appimageupdatetool to copy libraries that the Python bindings need to function. I'll open an issue about the naming.

[TheAssassin]

Re. 1: I did actually mean to fix it there, @TheLastProject.

Re. 2: AppImageUpdate is the UI application, and it ships with a Qt stack. python-appimageupdate should in our case be linked to the Qt shipped by PyQt, though. Therefore, it shouldn't increase Pext's AppImage file size that much. Please open an issue in python-appimageupdate's repository. I won't get to debugging this very soon, this way I won't forget it.

Re. 3: I see. Well, you could work around this by using the GitHub API, and using some Python JSON magic.

[TheAssassin]

I was trying to build the AppImage for testing, but appimagetool continuously crashes (wants to allocate a crazy amount of memory). Can you upload one for me to have a look please?

[TheLastProject]

The AppImage created by the Travis build of this branch is still online: https://transfer.sh/gCBAw/Pext-x86_64.AppImage.

[TheAssassin]

Thanks. It seems like, apart from the Python packages, boost is what bloats the package most. I don't use boost any more in favor of cffi. You should remove it, and check the AppImage's size again.
Also, usr/pkgs/cache seems like it can be removed, but occupies another 61 MiB (extracted).

[TheLastProject]

I got stuck in a update loop, so I figured I should call the path_to_new_file function. However, that seems to cause a segfault: https://transfer.sh/U8hkG/Pext-x86_64.AppImage (commit: ef18d03)

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: Unable to get the locale encoding
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f201616f740 (most recent call first):
Aborted (core dumped)

[TheAssassin]

Going to have a look.

By the way, I just tried the "Check for updates" button, and got a:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833)

[TheLastProject]

I have no clue what's up with the SSL error, it works for me on both Fedora and Ubuntu.

[TheAssassin]

I'm on Ubuntu xenial. But I extracted the AppImage, and ran the python binary in it directly with gdb. Might that cause this issue?

[TheLastProject]

Maybe, the Ubuntu I tried on is also Xenial.

[TheAssassin]

Well, it's quite obvious why this doesn't work. The AppRun script contains SSL related fixes that aren't applied when bypassing the script. I'll perform them in my shell by hand.

[TheAssassin]

Current version is an untagged development version, can only check for stable updates

How can I call the updater from the UI? Is there a parameter I can use to force a manual update?

[TheLastProject]

This is what happens if Pext isn't running inside the appimage. Just setting the variable APPIMAGE should work (but please make sure to set it correctly: https://github.com/Pext/Pext/compare/feature/appimageupdate#diff-f92d0e21ecdc5919116ff63380b7dbceR1357)

[TheAssassin]

Seems to work.

Could it be that you are starting a thread for every progress update? This slows down the process quite much when running in gdb.

[TheAssassin]

Seems like debugging the Python process with GDB isn't quite possible. Maybe we can try to add a unit test to python-appimageupdate?

[TheLastProject]

I am indeed using a thread for each progress update check, I'll try to remember to simplify that in the future, I forgot I can safely sleep in a second thread: https://github.com/Pext/Pext/compare/feature/appimageupdate#diff-f92d0e21ecdc5919116ff63380b7dbceR2630

[TheLastProject]

Unit testing python-appimageupdate sounds good too. Whatever gets this fixed :)

[TheAssassin]

The idea is to add a unit test that triggers this bug. It's already unit tested. In fact, its unit tests have been used to discover bugs in AppImageUpdate (which lacks unit tests at the moment).

[TheLastProject]

In that case, my best bet is that it relates to running an AppImage named like Pext-x86_64(1).AppImage while Pext-x86_64.AppImage exists and then updating. I haven't been able to test more to really see if that's related (it's just where I first saw the bug), but it seems a good place to start.

[TheAssassin]

How sure are you that the segfault occurs when calling path_to_new_file?

[TheLastProject]

Not completely, but ef18d03 is definitely the commit where it starts breaking. I'm not sure how else I'm supposed to possibly switch to the new AppImage after updating?

[TheAssassin]

The workflow seems alright, that's why I'm asking.

[TheAssassin]

I can't reproduce the bug in my unit tests. I'll build an AppImage in a VM now, and will see whether it'll work.

[TheLastProject]

Bump, just to make sure you're still aware of this issue (no rush!) :)

[TheAssassin]

I am, @TheLastProject, but unfortunately I am busy with AppImageLauncher and linuxdeploy at the moment. I will increase the prio of this on my TODO list, though.

[TheAssassin]

Been working a bit on python-appimageupdate recently. See also TheAssassin/python-appimageupdate#2 (comment).

Regarding:

In that case, my best bet is that it relates to running an AppImage named like Pext-x86_64(1).AppImage while Pext-x86_64.AppImage exists and then updating. I haven't been able to test more to really see if that's related (it's just where I first saw the bug), but it seems a good place to start.

AppImageUpdate would rename such an existing file to <name>.zs-old. This wouldn't happen when calling path_to_new_file(), but during the regular update process.

I'm going to look into this again now.

[TheAssassin]

Cannot build an AppImage at the moment due to:

Deleting pre-existing .DirIcon
Creating .DirIcon symlink based on information from desktop file
AppStream upstream metadata found in usr/share/metainfo/pext.appdata.xml
Trying to validate AppStream information with the appstreamcli tool
In case of issues, please refer to https://github.com/ximion/appstream

(appstreamcli:25761): GLib-ERROR **: gmem.c:103: failed to allocate 18446744073700670033 bytes
Trace/breakpoint trap (core dumped)
Failed to validate AppStream information with appstreamcli

@TheLastProject did you ever encounter this issue? At first I thought this was an issue in appimagetool, but apparently appstreamcli is crashing.

[TheLastProject]

Well, I do all my builds on Travis for AppImage, and those work just fine for me.

[TheLastProject]

I seem to be unable to get this to work at all with linuxdeploy:

https://travis-ci.org/Pext/Pext/jobs/451022936

[conda/stdout]     build/_appimageupdate.cpp:495:29: fatal error: appimage/update.h: No such file or directory
[conda/stdout]      #include "appimage/update.h"
[conda/stdout]                                  ^
[conda/stdout]     compilation terminated.
[conda/stdout]     error: command 'gcc' failed with exit status 1
[conda/stdout]     Running setup.py install for appimageupdate: finished with status 'error'