Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request: raw keyboard events #1523

Open
MagicalTux opened this issue Nov 12, 2020 · 12 comments
Open

Request: raw keyboard events #1523

MagicalTux opened this issue Nov 12, 2020 · 12 comments
Labels
Keyboard Items for keyboard control

Comments

@MagicalTux
Copy link
Contributor

A widget should be able to capture raw keyboard events, including keydown/keyup events.

For example the GameBoy emulator at github.com/andydotxyz/fynegameboy will check if a key repeats within 200ms to know if it's being held, which is not a good thing.

Instead there should be a way to access raw events where the Device allows it, and this process should be nothing more than a fallback.

@andydotxyz
Copy link
Member

Apologies but the code you linked is not the Fyne GUI but a telnet adapter.
See instead https://github.com/andydotxyz/fynegameboy/blob/2d31152a860878e3ae63ba2b8929ed94e36650a1/fyne/driver.go#L266 where the key up and key down events are used to get the hardware events you describe.

@MagicalTux
Copy link
Contributor Author

Great this is working. Now I just need to create a table to convert the fyne keycode name back to a scancode.

Some keys are not supported on 105 keys European keyboard or 109 keys Japanese keyboards, any chance the raw scancode could simply be added to the event and it be triggered even if the scancode wasn't found in the name table?

@andydotxyz
Copy link
Member

We don't want to expose the scancode as that is so hardware dependent - there are a bunch of tables inside the GLFW driver (internal/driver/glfw/) that converts codes / names to named keys in the Fyne API.
Are we missing the key names, or are we not managing to look them up correctly?

@MagicalTux
Copy link
Contributor Author

I think some of the key names are missing, but also the keypad keys and keyboard keys are mixed together and impossible to tell apart (for example French keyboards require to use Shift to access digits on the keyboard side).

Maybe there is a way to access directly glfw when on desktop and bypass fyne?

@andydotxyz
Copy link
Member

I'm not quite sure why getting access to the keyboard scancodes is more desirable than fixing the Fyne code.
I would like to understand this more so we can properly fix the issue.
We want to avoid exposing scancodes as much as possible because the hardware aspect is only relevant on desktop apps - but most apps just want to know if "7" was pressed, or the Enter key, for example - where scancodes would be a distraction as both can be triggered by 2 or 3 different codes...

@MagicalTux
Copy link
Contributor Author

Yes, it's OK for most cases to just know 7 was pressed, as long as you're in the US or UK. Most European countries do not use the same keymap and keys which are equal on us keymaps aren't equal anymore.

The missing keys are (lowercase followed by uppercase if any):

  • ²
  • :/

I do not know the appropriate scancodes or values in glfw for those keys however.

Adding the missing keys is fine, but I do not want to change fyne's behavior in terms of handling digits as some applications may be relying on this, hence a way to access glfw directly might be more suitable in my case, and others in the future may want to hack further than what's possible.

For example a Canvas can be tested & instanciated as a desktop.Canvas, is there anything similar for keyboard?

@andydotxyz
Copy link
Member

Most European countries do not use the same keymap and keys which are equal on us keymaps aren't equal anymore.

Indeed, this is why we want to not have developers worry about scancodes and keymaps. Our APIs are focused on the user intent rather than the implementation where possible :).

Adding the missing keys is fine, but I do not want to change fyne's behavior in terms of handling digits

I don't think that adding new key mappings will break behaviour.

@MagicalTux
Copy link
Contributor Author

MagicalTux commented Nov 12, 2020

I don't think that adding new key mappings will break behaviour.

I fully agree, however this is more complex than it looks like.

For example on my keyboard (French) when I press & I receive a Key1 KeyEvent. That's because & is the lowercase of 1 on the keyboard side. However if I press A I do indeed get KeyA. The behavior as it is today is not consistent and it's difficult to make something that will both work for games (need to handle keys pressed separately from modifiers as these typically can trigger things, such as run, walk, crouch, etc) while being friendly enough (a US developer will set KeyA to strafe left by default, but on a azerty keyboard that key won't be at the right place and it'll mess up the whole thing).

User intent can be different between regular typing (I want to press A) or gaming (I want to press the key left of S). I grew in France and spent years dealing with games where I had to press W to move forward (W and Z being inverted on French keyboards). You get used to it.

Another example: FyneGameboy has hardcoded the X and Z keys for A/B. On my keyboard the key next to X is W, the Z key is at the left of E on the top row. This is extremely awkward to use for anyone with a non-qwerty keyboard and non configurable in the case of FyneGameboy.

Now, providing the scancode can be a pain. It could be XT, AT or USB, or whatever appears in the future. Maybe it could all be converted to a single standard (fyne standard, even?) or anything. Having in the KeyEvent both the logical (character or anything) and physical information of the key would make sense, a lot of frameworks out there do that. The developer can then choose to use whatever makes sense for him.

It also helps avoiding breaking the current behavior.

@andydotxyz
Copy link
Member

You're certainly right this is a complex issue :)

Now, providing the scancode can be a pain. It could be XT, AT or USB, or whatever appears in the future. Maybe it could all be converted to a single standard (fyne standard, even?) or anything.

I guess that's what I was considering as well. I know that many toolkits expose the hardware side of this but I don't think many toolkits have succeeding in abstracting correctly the hardware details so apps can be built once and run correctly on all platforms. I don't know the right way to get the best of all worlds.

Another challenge we have is shortcuts - we have recently discovered that keyboards that have certain character combinations will not correctly map to (for example) ctrl-A because their "A" key is not actually an A, but on some OS they map the different letter to certain keys for internationalisation workarounds.

@MagicalTux
Copy link
Contributor Author

MagicalTux commented Nov 13, 2020

Another option for multiplatform UI is Flutter by Google. Their use for raw keyboard events on all platform rely on USB scancodes (see flutter PhysicalKeyboardKey class documentation).

I don't think it matters much which standard is used however as long as constants are made available.

In the meantime if there is a way to access glfw/gomobile directly via Fyne (some interface{} thing somewhere? or maybe I can instantiate these directly? To be quite honest I haven't looked into it yet since I've been working on other parts but I'm getting closer to needing this) I may be able to get more information, and implement something that works and can be used to then improve fyne - or not if it's too specific, but at least it'll allow me to move forward.

@andydotxyz
Copy link
Member

Good illustration, thanks. I think this is what fyne.Key is attempting to do - refer to the source of an event be it physical, software or simulated. The result of key presses is the TypedRune and TypedKey (the first is for visible chars, the second for other keys) so we already have an abstraction built in. If physical keys are missing from the list we should add them. Most are in the main package, but some that appear only on desktop (such as "Super") are in the driver/desktop package.

@andydotxyz andydotxyz added the Keyboard Items for keyboard control label Nov 17, 2020
@andydotxyz
Copy link
Member

ScanCode now added, we can discuss next how a KeyPosition field or PositionName() function could add a further "abstraction".

@andydotxyz andydotxyz added this to the Aberlour (2.1) milestone Aug 15, 2021
@andydotxyz andydotxyz removed this from the Fixes (v2.1.x) milestone Jun 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Keyboard Items for keyboard control
Projects
None yet
Development

No branches or pull requests

2 participants