Abstract mouse/window/keyboard interfaces

[sebastiansam55]

TLDR

With my recent discovery that I can poll the mouse location and interact with windows easily (on gnome shell at least) it opens the path toward (hopefully) easily port AutoKey over to full support on Wayland with minimal to no loss of major features

STATUS

Abstracted Window interface, working with AutoKey gnome shell extension to get autokey working on Wayland/gnome distros (Ubuntu 22.04 for ex).

The longer term plan is to have things setup internally so that we can easily add additional interfaces for different display servers and different desktop environments.

https://github.com/sebastiansam55/autokey-gnome-extension

This GNOME shell extension is a fork of one that provides a number of dbus methods that allow us to interact with windows under wayland. We can get the active window information and move/resize/close/maximize/minimize.

It will also allow us to poll the location of the mouse pointer, with this we can use uinput to move the mouse in x y until it is approximately in the correct location. TBD How hard that will actually be to make look smooth/fast.

Unfortunately Wayland does not allow the pointer to be warped.

TODO

• Add checking that Autokey Gnome Extension is properly installed
• Add config file/settings option to set window interface
• Add additional dbus functions to extension for mouse interactions Wayland does not allow anything other than polling location
• Abstract mouse interface
• implement mouse movement via uinput events
• implement mouse clicks
• finalize and publish gnome shell extension
• update tests for window and other files
• kde has plasma extensions but I need to do more research on it.

Something needs to be done about the Button and Key classes in model. They need to be abstracted in someway that will allow Key.LEFT to correspond to the correct scan code/key code per the active keyboard system.

New

uinput events

For hotkeys, under uinput we should be able to capture any event, so everything from foot peddles to mouse buttons, gamepads and keyboards we should be able to record and listen for as an event for a hotkey. Very cool.

left/right modifiers

I am also going to strive to bring left and right differention to both the hotkey modifiers and the keyboard send system, have to either bring support for that to x11 keyboard or make it backwards compatible. Setting the generic version to left makes the most sense.

IE: For x11 there is currently only Key.SHIFT with no diffentiotion between left/right. For uinput I'll be trying to set it to respect all Key.LSHIFT Key.RSHIFT and Key.SHIFT. the last would be sent as left shift

internal changes

Mouse, Keyboard, window interfaces are all going to be separated out to be independent of each other, with this done it will be easier to implement support for Wayland.

Scenarios:

GNOME/Wayland

• windows via GNOME extension
• keyboard
  • read
  • write
• mouse
  • read via GNOME extensions
  • write

KDE/wayland

• windows via plasma/KDE extension?
• keyboard
  • read
  • write
• mouse
  • read
  • write

[GNOME / KDE]/x11

• windows
• keyboard
  • read
  • write
• mouse
  • read
  • write

CHANGES

window is going to change scripting API wise, will be breaking change, ultimately it's my fault for not being forward looking when adding those methods, they were tightly specific to x11

mouse will likely evolve, right now there's really only move and click but you don't send a click or don't send a move if you don't want to do both, would like to split those out

[josephj11]

This is quite encouraging although I don't pretend to understand the specifics. Let us know (on Gitter, etc.) if there's anything we can do to help.