Packaging of Autokey

[sebastiansam55]

I'd like to start a concerted effort to understand, document and automate releases for autokey, I think decreasing the effort/friction needed to push out an update would greatly benefit autokey and the development lifecycle.

At minimum I know we have;

• Ubuntu Packages
• AUR repository packages
• PIP package
• Installing from Source

Last time I checked BlueDrink9 implemented a github actions workflow for updating the pip package which looks to still be in place.

So we need to document/automate;

• Generating .deb package
• How the AUR system works

Anyone have any idea how those systems work? I think we've gotten issues before from the person who maintains our debian package?

[Elliria]

I can't answer all of that, but I can give you the notes I've put together so far from what I've been able to figure out about how our process works and what will need to be done to build AutoKey. This is a work in progress and you're getting my sloppy copy here, but maybe some of it will be useful:

Our Git Flow workflow:

1. Use this structure for the repository:
   • master has been deployed as the current release and can currently be deployed.
   • develop is branched off of master and is intended to be used for active development.
   • beta is branched off of master and is intended to be used for all commits from develop that we want to keep.
   • Features are worked on or created on feature branches off of the develop branch and are merged into the develop branch when each feature is finished.
   • Hotfixes (critical changes) are done on hotfix branches off of the master or develop branch that are merged into the master or develop branch when each hotfix is finished.
2. Do the development on feature branches off of the develop branch or do development on the develop branch in an AutoKey fork.
3. When you're satisfied with the develop branch and are ready to create a beta release:
   1. Announce a development freeze to the developers.
   2. Update the CHANGELOG.rst file on the develop branch with the new beta release version and a new, empty Develop section at the top.
   3. Update the Version line in the PKG-INFO file on the develop branch, if applicable.
   4. Change to the beta branch: git checkout beta
   5. Ensure that the beta and master branches are synchronized by doing a hard reset of beta'' to master, removing all staged and unstaged changes from beta, resolving conflicts in favor of master, and updating the beta tip (its most recent commit) to the commit at the tip of the master branch, making beta exactly the same as master**: **git reset --hard master''
   6. Use cherry-pick or merge to bring any or all commits that we want to include from develop onto beta.
   7. Tag the beta branch with a beta release version number.
   8. Push the change to the remote repository: git push
   9. Put the beta files onto GitHub for download.
   10. Announce the beta for testing by others.
4. Pick one:
   • If you're not happy with beta, return to to step 2 above and continue development.
   • Or, if you are happy with beta, create an AutoKey release:
     1. Change to the master branch: git checkout master
     2. Back up the master branch in your stash so you can recover if it all goes badly: git stash
     3. ????? Wipe out all differences in the working tree on the master branch (ignore everything up to now) (I believe this gets rid of the X commits ahead and X commits behind messages so you'll start clean): git reset --hard
     4. Merge beta into master, favoring beta for all conflicts (make master an identical copy of beta):
        1. Change to the master branch: git checkout master
        2. Merge beta into master using beta's's history (resolving all conflicts in favor of beta): git merge -s theirs beta
        3. Enter your commit message and save when prompted.
     5. Update the VERSION line in the common.py file on the master branch with a new AutoKey version.
     6. Create an AutoKey release by tagging the master branch with a version number on git or GitHub.
     7. Update the VERSION line in the common.py file on the develop branch with a new beta version.
     8. ????? Do a hard reset to wipe out all differences in the working tree on the master branch (ignore everything up to now): git reset --hard
     9. ????? Do a hard reset to wipe out all differences in the working tree on the develop branch (ignore everything up to now): git reset --hard
     10. Push the changes to the remote repository:
         1. Change to the beta branch: git checkout beta
         2. Push the changes to the remote remotisoty:git push
         3. Change to the develop branch: git checkout develop
         4. Push the changes to the remote remotisoty:git push
         5. Change to the master branch: git checkout master
         6. Push the changes to the remote remotisoty:git push
     11. Put the AutoKey files onto GitHub for download.
     12. Announce the new AutoKey release to our contacts at the various distributions.
     13. Return to step 2 to begin work on the next release.

Build AutoKey

If you've got what seems like a good working copy of the code, it's time to build it into files that can be installed on a machine so you can try it out and see if it works.

TODO: Find out if there's any difference between doing the build from the zip file, the way we do below, and doing it from a cloned copy of the fork.
TODO: An alternative to the following instructions may be to run autokey/debian/build.sh to build the .deb files, but I'm thinking that will be run automatically when following the steps above. I'll update this once I do a successful build.

1. Open your virtual machine.
2. Open a browser inside the virtual machine.
3. Go to the AutoKey repository's main page on GitHub.
4. Choose the branch that you'd like to build.
5. Click the Code button.
6. Click Download ZIP from the drop-down menu.
7. Extract the zip file, creating a directory that contains the contents of the repository.
8. Open that directory in a terminal window.
9. Pick one:

• Install AutoKey:

  pip3 install .

• Build AutoKey:
  1. Build the packages in the parent directory of AutoKey:

     dpkg-buildpackage -us -uc

  2. Test the build:
     1. Change to the parent directory:

        cd ..

  3. Install the GTK version:

     sudo dpkg --install autokey-gtk_<version>.deb autokey-common_<version>.deb
     ```					```

  4. Install the Qt version:

     sudo dpkg --install autokey-qt_<version>.deb autokey-common_<version>.deb

  5. Install any missing dependencies:

     sudo apt install -f

  6. See if the build works, where it puts AutoKey, and any messages/errors it generates.
  7. If everything works, copy the .deb files it produced to a clean virtual machine and repeat steps c through f to see if they install and work properly there.
  8. ??? I GOT THIS FROM THE DEBIAN WEBSITE ??? If everything is working and the build is satisfactory, sign the .dsc and .changes files with your private GPG key using the debsign command. You'll need to enter your secret pass-phrase twice.

[josephj11]

Nice. Under Beta, I would change "keep" to "include in the next release".

[Elliria]

Thanks. I did that. I also updated step 2.

[Elliria]

The Packaging wiki page may also be of use.

[josephj11]

Getting this down will be a major infrastructure enhancement.

AFAIK, the only packaging we do is our local debs and tarballs - which I believe are already automatically generated when a release tag is made and the PyPI/pip stuff. (IDK how that stuff works.)

Although there are two aur packages. I don't think AutoKey makes them. Bluedrink9 was listed as their maintainer last I looked.

There are other packages. RPM and DNF and whatever OpenSuse does come to mind, but they are all done by distribution team packagers (which is also the case for Debian who don't use ours). I think there's even a BSD package.

I'm pretty sure distros want full control of the packages they include, so our efforts should be to ping them when a new release occurs.

Two Debian packagers have communicated with us in the past, but neither are still involved with AutoKey. (Luke Faarone (probably misspelled) and Anthony Fok) Luke was only involved with the original branch of the project. Now the link shows "Debian packaging team" or some similar generic designation.

[Elliria]

Agreed on the infrastructure enhancement. Once these steps are solid, they can go into the CONTRIBUTORS.rst file.

PyPI steps are on the Packaging wiki page, although those could use an update to make them easier to use. There are other packaging steps on the same page.

BlueDrink is the Arch maintainer, the Debian Python Team is the Debian maintainer, raghusiddarth is the Fedora maintainer, and although we have OpenSuse users, there's no official package for them.

I like the idea of pinging distributions when there's a new release. Would that be done right before a release or right after? If it's before, it can be added to issue #905, which can be renamed as just Pre-release so that it can hold a task list. If it's after, we could create a Post-release issue for those sorts of tasks.

[josephj11]

I favor post release because we could announce it and then get held up and not release for an indeterminate amount of time.

If it's released, they can just do their thing without waiting for anything. They tend to package when they get around to it. So it's not like they're waiting for us and ready to spring into action. I'm sure they are much more responsive to really large projects, but we're not in that league.

It's possible that @BlueDrink9 will also be stepping down on the arch side. If he does, it's an arch issue, not an AutoKey issue.

[Elliria]

Okay, that sounds good. I've added it to my notes and will update the steps above with it and then some (more edits have been done).

[Elliria]

The Our Git Flow workflow: section above has been updated with lots of changes.

[josephj11]

Took a look at the GitHub Cheatsheet page you built. Impressive. You might publish it someday.

A few comments:

Do you have to use a VM for everything you say to use it for? I never got the hang of those, but I'm sure they're much better than when I was trying to get them to work years ago.

AFAIK, there are a number of VM systems other than Virtualbox. Is there any reason to only use Virtualbox?

If you're not already familiar with the apt-file command, it's in the standard repos (not installed by default). It's quite handy for figuring out what package you need to install to get a particular file or program.

It appears to be written in Perl. I noticed this when I was looking at the man page and saw a regex written in what I believe is the PCRE dialect. I have never needed to use a regex with it, but it's good to know.

The command has a lot of powerful options, but I have never needed any of them, so don't be put off by the apparent complexity.

[Elliria]

A VM isn't necessary, but I include it for safety so that you don't mess up your system or your existing AutoKey installation or files with any experimentation. If everything is done in a VM, it's isolated. Any time I do anything on a computer that involves trying something in which the outcome of the attempt is unknown or uncertain, it happens in a VM. A lot of things can go gloriously wrong when doing development, so you're better off safe than sorry.

Instructions for getting VirtualBox and setting it up and using it are on that GitHub Cheat Sheet page, but that page is mighty busy, so I also created a separate VirtualBox Installation and Use Gist that gives a brief walk-through of getting and using VirtualBox.

There's no reason to use VirtualBox over any other virtual-machine program. My reason for using it is that it was the only one I could figure out how to work when I tried them all years ago, so I've stuck with it.

I hadn't heard of apt-file and I notice that it's not installed by default, but it's interesting. It apparently lists the files that a package contains.

[Elliria]

The steps for Our Git Flow workflow above have been updated again. There are three sections where there are questions marks (?????) indicating that I'm unsure of those. As always, input, feedback, corrections, and/or suggestions are welcomed.

[BlueDrink9]

Probably too much confusion in that for me to totally clarify, but most of the ??? sections don't look that important so I'll ignore them.

Arch:

beta package auto-installs from the latest beta branch.
The master release is trivial to update by the maintainer for this project, (atm me): just update which commit it points at.

CI

Building the CI was intended to largely make releases trivial, I now realise there are several steps I took for granted. They should be clear to someone that is skilled in git.

The CI will handle pip releases, and create debs to attach to the github page.

debian

Automating the actual debian release was something I found far too difficult; I would definitely advise getting help there.