External scripts called with absolute path name do not have API access

[elydpg]

AutoKey is a Xorg application and will not function in a Wayland session. Do you use Xorg (X11) or Wayland?

Xorg

Has this issue already been reported?

• I have searched through the existing issues.

Is this a question rather than an issue?

• This is not a question.

What type of issue is this?

Bug

Choose one or more terms that describe this issue:

• autokey triggers
• autokey-gtk
• autokey-qt
• beta
• bug
• critical
• development
• documentation
• enhancement
• installation/configuration
• phrase expansion
• scripting
• technical debt
• user interface

Other terms that describe this issue if not provided above:

No response

Which Linux distribution did you use?

Kubuntu 23.10

Which AutoKey GUI did you use?

Qt

Which AutoKey version did you use?

0.96.0

How did you install AutoKey?

From the .deb file from git

Can you briefly describe the issue?

Pretty much the title. For some reason calling an external script using the absolute path name does not work properly; i.e. autokey API calls aren't available for some reason. Calling using the script description works normally.

Can the issue be reproduced?

Always

What are the steps to reproduce the issue?

1. Create the following scripts in a folder called "Scripts" within AutoKey:

CallExternal:

engine.run_script("~/.config/autokey/data/Scripts/External.py")

External:

if not store.has_key("runs"):
    # Create the value on the first run of the script
    store.set_value("runs", 1)
else:
    # Otherwise, get the current value and increment it
    cur = store.get_value("runs")
    store.set_value("runs", cur + 1)

dialog.info_dialog(message="I've been run %d times!" % store.get_value("runs"))

2. Bind the CallExternal script to the F1 key, for example
3. Save and press the F1 key

You should see the following error message pop up:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/autokey/service.py", line 530, in _execute
    exec(compiled_code, scope)
  File "/home/<user>/.config/autokey/data/Scripts/CallExternal.py", line 2, in <module>
    engine.run_script("~/.config/autokey/data/Scripts/External.py")
  File "/usr/lib/python3/dist-packages/autokey/scripting/engine.py", line 362, in run_script
    self.runner.run_subscript(path)
  File "/usr/lib/python3/dist-packages/autokey/service.py", line 580, in run_subscript
    exec(compiled_code, scope)
  File "/home/<user>/.config/autokey/data/Scripts/External.py", line 3, in <module>
    if not store.has_key("runs"):
           ^^^^^
NameError: name 'store' is not defined

Which suggests the API isn't being imported properly in this case.

However, changing the CallExternal script to read

engine.run_script("External")

And then pressing F1 again reveals the script working normally.

What should have happened?

No response

What actually happened?

No response

Do you have screenshots?

No response

Can you provide the output of the AutoKey command?

No response

Anything else?

No response

[josephj11]

Welcome to the AutoKey community, @elydpg !

This is a deficiency (making fixing it an enhancement), not a bug. It's undefined behavior. It is a variation of #248 and that issue shows one or more possible workarounds.

This can possibly also be worked around by having your external script launched using autokey-run (in the original call or by calling a normal script/program first).

I am not aware of anyone implementing something like this before, so please let us know what you accomplish. There's always room in our wiki for new articles and example scripts.

As always, feel free to join us over on Gitter to discuss this further or to ask for help implementing it.

[elydpg]

I see, thanks for letting me know! I keep my scripts in a ~/bin/macros folder so that's why I wanted to be able to call them using absolute path names. Though it doesn't seem to make a difference whether I move them into the AutoKey folder or not (as illustrated by the reproduction steps), absolute paths do not work. Can I ask why this isn't a bug though? I don't fully understand why running the scripts using descriptions versus paths would make a difference to the availability of the API calls.

[josephj11]

OK. The gory details (and my opinions as a very long time user and project member).

First, just in case something stupid simple will work:

The following should work fine, but always make a full backup of the AutoKey configuration and your external script directories first. Given the amount of work that often goes into using AutoKey creatively, you should have these backed up regularly anyway.

On the left of the Main AutoKey window is a file tree. When you select that panel, you can then click on New in the main menu (with or without first clicking on the top level folder. You just need to be sure you haven't selected a subfolder).

New->Folder->Create Elsewhere

This will let you define a folder anywhere such as where your scripts are now. Maybe that will solve your immediate problem without the rocket science below. However, when you do this you will see a warning message which you need to read carefully and understand because if you don't, AutoKey may bite you in the future. This is not meant to deter you. It's just things you have to know.

I have had all of my actions (scripts and phrases) in an external directory sub-tree for years and much prefer it that way. Notably, I can uninstall or even purge AutoKey and my actions are still there. If that directory happens to be a symlink, I can change the entire set of actions to another one just by pointing the symlink elsewhere (preferably, when AutoKey isn't running!). The external directory can even be in a directory used by Dropbox or Mega... so it can be accessed by different instances of AutoKey on other machines or partitions (not concurrently! and not without a few small side effects).

---

The way AutoKey works with scripts (I'm clueless on the internals) is that when you run a normal script within AutoKey, it exposes the API modules for the script to use without you doing anything additional. When you run scripts with components imported from the Modules directory or eternal scripts from anywhere else (because you know some Python), the API modules are not exposed to those components or external scripts, so the API calls don't work in them. This is a major deficiency.

I take the position that if we don't say something works, then when it doesn't work it's usually not a bug. To me a bug is something that works incorrectly.

In this case, people like you and me expect modules/functions in the Modules directory to have API access. That's a pretty fine line between deficiency and bug. However, if you have functions that don't require the API, they work fine. The code works fine as far as it goes. It just stops short of what we expected it to do.

I pushed for this to be fixed for a long time. A couple of talented developers took a crack at it, but ended up mumbling some things about the nuances of low level Python that they didn't understand. So, apparently, it's not the easy fix it appears to be from the outside.

This issue made me think about this problem again and I wrote a small (unpublished) article about it. I may have come up with some small improvements to the workarounds discussed in the issue referenced above. However, I don't have time anytime soon to write the test code to see if my ideas actually work. That's why the article is unpublished. I don't know a lot of Python, so I do things slowly.

Here's a link to the first draft of the article.

For those finding this later, this is a temporary link not guaranteed to last. If/when the article is ready for prime time, I will add it to our wiki.

I think your best bet is to start with the last suggestion and see if autokey-run will do what you want.

If you don't see how to implement an option, I can talk you through what I think will work. But not here. I didn't have time to include those details in the first draft. To pursue this, come over to Gitter where we can discuss things freely and maybe even get help from others there.

[elydpg]

Wow, thanks so much for the detailed explanation! The solution I went with with was even simpler though. I just symlinked the external scripts into AutoKey's default folder and it ran them no problem. I was able to call scripts by description and have it work the way I want it to ☺️.

Hopefully this gets resolved at some point, as it's definitely a bit unintuitive that whether or not the API gets auto imported depends on how the external script is called...

[josephj11]

Never thought of trying that. It was too simple! :)

@elydpg You should still start to try the top method of Create elsewhere because I strongly suspect that the warning message it displays also applies to what you have done. (Do it just to see the warning message.)

[elydpg]

I think it's this warning right?

Not sure it applies since AutoKey would just delete/move the symlinks in that case no?

[josephj11]

I wouldn't bet on that although you may be right. If your original directory contained non-AutoKey action files and you moved it from within AutoKey, all those files would probably be deleted.

Long ago, there was a bug such that if you added an external directory to AutoKey, it would create that directory with the side effect of deleting everything that was in it before. This was quite unpleasant and was quickly fixed once it was discovered. It left me being extra careful to back things up before trying anything like that!