Add Gnome Extension Window interface, left-right modifier distinction changes

[sebastiansam55]

Refactor interface.py

Changes to interface.py to use decorator functionality instead of __enqueue to make the code more readable and reduce overall number of methods.

Window Interface

Pulls out the window methods from XInterfaceBase and adds the GnomeExtensionWindowInterface. still in the early testing stages of these changes, if anyone else would like to test (please!) you'll currently need to edit iomediator.py: 69 and file the if statement so that self.windowInterface is an instance of GnomeExtensionWindowInterface.

I use this day to day currently and it works without issue (admittedly I don't use the window api much in day to day use).

There is one minor issue related to gnome-screenshot -a, it seems like because something funky is happening with the input focus it throws an error.

Left-Right Modifier keys

Adds a distinction between the left and right modifier keys.

As noted in the changelog this should probably be considered a breaking change, before both left ctrl and right ctrl were just seen as "ctrl", now they're seen as distinct. The modifier will check for if you're doing all left or all right modifiers but it will not try to "mix and match"

Currently support for this only exists in GTK, I will look into the Qt side sooner or later. You should be able to manually make changes to the config files to get the left/right distinction when using Qt front end tho.

UInput Technical Details

Hotkeys now technically fire on keyrelease instead of keypress

To be clear, this is only a uinput interface change, X11 should still operate as it always has
Technical reasons why, but especially with phrase expansions there are weird issues with the keys still being pressed while the expansion is executing, it was easiest to just wait for the key release.

UInput Device Selection

Currently, just searches the device descriptions for "mouse" or "keyboard" respectively.

Need to integrate in the setting file and the UI.

Keyboard exclusive grab

Currently, AutoKey grabs the entire keyboard input wise and passes the input via the UInput device that it creates.

Using exclusive grab would also allow keyboard remapping, I have similar projects on my personal github, I'll eventually try to bring that to autokey as well.

GTK

Qt

I'll look into adding another row to the QT frontend that would represent the "either" option

Documentation changes

Updated the documentation/RST to remove "gremlins" (quote symbols that aren't ascii) and fix RST warnings when sphinx-build runs.

Known Issues

Special keys like "F13" are translated as <code193> for the purposes of saving hotkeys where 193 is the x11 keycode. 193 does not map to anything uinput wise Not sure if it is worth setting up something to translate this.

Looks like there may currently (05/21/2023) be an issue with clipboard phrase expansions.

Installing this branch from zero on ubuntu 22.04 for testing;

sudo apt install autokey-gtk
sudo apt install python3-evdev
sudo usermod -a -G input $USER
# have to reboot at this point

sudo apt install gnome-shell-extension-manager
git clone https://github.com/sebastiansam55/autokey-gnome-extension
cd autokey-gnome-extension
gnome-extensions install autokey-gnome-extension@sebastiansam55.shell-extension.zip
# have to logout and back into wayland

# check using the install gnome-shell-extension-manager that the autokey extension is enabled

Ubuntu 22.04 (at least, likely most other distros?) require a file in udev;
/etc/udev/rules.d/10-autokey.rules

# uncomment in case of:
#     evdev.uinput.UInputError: "/dev/uinput" cannot be opened for writing
# see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=827240
KERNEL=="uinput", SUBSYSTEM=="misc", OPTIONS+="static_node=uinput", TAG+="uaccess", GROUP="input", MODE="0660"

Now you can run autokey and it should work

git clone https://github.com/autokey/autokey --branch develop-decorator autokey-testing
cd autokey-testing/lib
python3 -m autokey.gtkui -lc

TODO:

• Qt "universal" toggle hotkey gui
• Device selection GUI?
• Device selection memory in settings file
• Consider how to handle "<code" things
• more complete testing, there are a lot of features in autokey.

[sebastiansam55]

@Elliria and @josephj11 if y'all could try to test this as well that would be awesome, like I said I'm currently using the branch day to day but my day to day usage doesn't involve major parts of the changes here.

[BlueDrink9]

Don't have time to look fully, but I'll try to review this as I can. Changing to decorators was something I was intending to do ever since learning about them, so well done!

Suggest giving instructions for checking out this branch, since joe and elliria are still new on their git journey

[josephj11]

@BlueDrink9 You better tell us when you complete your degrees (regardless of whether that impacts your status at AutoKey) so we can celebrate with you. Best of luck!

[Elliria]

Good to see you again, @BlueDrink9.

Suggest giving instructions for checking out this branch, since joe and elliria are still new on their git journey

No worries. We've got it covered in the Trying out a clone of AutoKey entry in the wiki, complete with the Clone a repository branch section inside of it.

But thanks for mentioning it. For some reason, I was never notified of that message up there asking us to test it.

[BlueDrink9]

My main concern is showing you how to check out a pull request, specifically (a bit more involved, but not by much)

[Elliria]

Figured it out. You can follow these steps to test it out, too, @josephj11, and anyone else who would like to can feel free to jump in as well:

1. Open a terminal window in the directory in which you'd like to create the clone's subdirectory.
2. Clone the develop branch:

   git clone --branch develop --single-branch https://github.com/autokey/autokey.git
   

3. Change to the clone directory you just created:

   cd autokey
   

4. Fetch the pull request:

   git fetch origin pull/844/head:pull_844
   

5. Switch to the pull request's branch:

   git checkout pull_844
   

6. Change to the lib directory:

   cd lib
   

7. Run AutoKey in verbose mode with the main window opening by default with one of these commands:
   • GTK:

     python3 -m autokey.gtkui -lc
     

   • Qt:

     python3 -m autokey.qtui -lc
     

[josephj11]

I'm concerned about legacy user action triggers (and being able to select them in the future). The old/current behavior is to treat the left and right versions as identical. Shouldn't that still be an option going forward? What will happen to existing actions? Will they now only be triggered by one of the two keys making the other one no longer work?

This gets more confusing if, e.g., I would like a hotkey of Shift+Left Alt+... This "mixed mode" setting where shift is undifferentiated, but Alt is differentiated ought to be possible. It's not clear to me how the user could specify this when recording a new hotkey. There would have to be additional dialog element(s) to support this.

Maybe a checkbox next to each selected modifier key to specify whether it is one or both. If this was implemented, it would also suggest something like disallowing pressing both keys when the option is selected. E.g., if the user specifies "both" for Alt, then recording/setting Left Alt+Right Alt+... doesn't make any sense and might have unintended consequences.

This might not just be a personal preference issue. If one AutoKey configuration is deployed to multiple users (as I believe is already being done by some users), there may be invalidated muscle memories or even accessibility issues where one choice will not work for all these users.

[Elliria]

I agree with @josephj11 that these should be an addition to the original choices rather than a replacement for them. I took screenshots of both the current choices and the new ones that are being worked on in this PR:

Both are presented in six columns, so perhaps having three rows instead of two and just keeping the original row at the top would work. I also really like @josephj11's suggestion of checkboxes. That way, you could probably have a small selection of buttons and then just some little checkboxes beneath each one for left and right, either of which would override the default value of both that the originals have.

For testing purposes, I was limited in what I could do and am not certain I was able to test everything. This is simply because some of those keys don't exist nowadays and have to be emulated creatively. I did my best to do with within the powers of KDE's keyboard customization, but the right versions of several of them cannot adequately be tested. Also, the AltGr didn't respond for me and just brought up the print interface, probably because KDE intercepts that before AutoKey would have a chance to see it. I can do a trace if that would be helpful. Either way, other than that one, the ones I was able to test all worked in GTK and Qt.

[josephj11]

There appears to be a lot going on here.

Looks like you implemented the x11 test, but I don't see what it does when it fails.

Also looks like you accomodated undiffferentiated modifier keys along with the new differentiated ones.

Looks like the undifferentiated modifier keys haven't made it to the Qt front end yet. That doesn't hold up this PR, but it does seem like it should block releasing the feature.

Thanks for the great work.

[Elliria]

It looks like you've gotten a lot done. You're a machine, @sebastiansam55.

I'd like to test the window interface, but I don't understand this part: " if anyone else would like to test (please!) you'll currently need to edit iomediator.py: 69 and file the if statement so that self.windowInterface is an instance of GnomeExtensionWindowInterface." Any help with that would be appreciated.

I'm also going to give the GTK universal options as solid a test as possible, given that I don't have all of the necessary keys. It looks good, though. I'm curious, however, as to why you chose a toggle instead of a button for the universal controls. Both accomplish the same goal. It might look better and be less confusing for everything in that hotkey interface to be buttons.

[sebastiansam55]

notes on it are out of date. Going to push a major commit here soon, will update the PR notes after that.

[josephj11]

@Elliria I don't quite see how a button would work. When you record a hotkey, AutoKey will see which variant you pressed. How will it know you want both to work? And what if you want some keys differentiated and others not?

As an aside, I haven't explored it, but when I manually click on the modifier keys in that dialog, I have the impression that it doesn't work as expected.

[sebastiansam55]

@josephj11 and @Elliria

Should be "MVP" (minimum viable product) for wayland. It's to the point where I will probably start using wayland as a daily driver. It runs and does hotkeys/expansions that I use. Testers welcome. If someone could start a discussion thread for errors related to it, that might make more sense than managing it in this PR itself.

Some major caveats should come with testing, there are a lot of pitfalls, even once it's installed and configured. You (or autokey) can "softlock" your system. In order to provide "hotkey" grabbing I have the keyboard exclusively grabbed and am passing events through via the UInput device. This, in combination with the window information would allow us to match the "hotkey" grabbing that the X11 version can do. Currently not implemented.

But as it grabs the actual keyboard exclusively, if there is an error or exception it will probably lock up the keyboard, so you should keep a way to kill autokey available, whether that be SSH or closing the terminal window.

The way it's currently configured to select the keyboard and mouse relies on the device descriptions having "keyboard" or "mouse" in them. There's not a better way to do it automatically that I could find.

Will eventually add a GUI for selecting the input devices.

[Elliria]

@Elliria I don't quite see how a button would work. When you record a hotkey, AutoKey will see which variant you pressed. How will it know you want both to work? And what if you want some keys differentiated and others not?

No worries. It's fine as is.

As an aside, I haven't explored it, but when I manually click on the modifier keys in that dialog, I have the impression that it doesn't work as expected.

That has always happened to me with modifiers when setting AutoKey hotkeys, often making it necessary to press them more than once or to press the character key first and then the modifier, which will always work on the first try. No idea why. I don't use hotkeys, though, other than for testing.

[Elliria]

Should be "MVP" (minimum viable product) for wayland. It's to the point where I will probably start using wayland as a daily driver. It runs and does hotkeys/expansions that I use. Testers welcome. If someone could start a discussion thread for errors related to it, that might make more sense than managing it in this PR itself.

I'll try it once you've updated those notes above on what we need to do.

But as it grabs the actual keyboard exclusively, if there is an error or exception it will probably lock up the keyboard, so you should keep a way to kill autokey available, whether that be SSH or closing the terminal window.

There's a script on this Reddit page that might work the same way as our beloved xkill command in a clickable shortcut on the taskbar. Someone on another page recommended using the system monitor to do it.

Will eventually add a GUI for selecting the input devices.

That sounds good. You're amazing, @sebastiansam55.

[josephj11]

I have a lot of questions. Starting a discusion. #866

[josephj11]

This is a substantial piece of work!

Other than replacing a ton of underscores with dashes, most of this is over my head.

I'm a bit confused about how this fits in our release cycle. I was thinking that the next release would be 0.96.1 with small changes like #600 ... This looks more like it would go into 0.97.0.

@Elliria was developing documentation of our release cycle that seemed to imply that develop would only contain code for the next release with everything else staying in feature branches until it was ready for release. How are you viewing this process/methodology?