Enable finding of undisplayed menu items

[bernd-wechner]

I found with our UIA app that:

config_item = menu.item_by_path("File -> Configuration", False)

did not work. Pywinauto crashes in fact. Traced to the fact that menu.items() does not see the File menu's items until the File menu is clicked and they are displayed. That is the UIA controls don't exist until the they are displayed.

This line of code works with this patch, and I can find and then click the Configuration item on the File menu using:

config_item.click_input()

and all is good.

There is a strange behaviour I don't understand that the cacheing works around. Essentially in MenuWrapper.item_by_path() is called once and clicking the menu works, that is in MenuItemWrapper.items(), self.children() contains the menu items after it's clicked.

MenuItemWrapper.items() is called a second time in MenuWrapper.item_by_path() though, via:

menu = next_level_menu(menu, menu_items[i], items_cnt == i + 1)

which calls self._sub_item_by_text which again fetches the menu items with items = menu.items().

On this second call I find that the self.children() in MenuItemWrapper.items() still has all the children, but mysteriously they have lost all their element_info, they all have no names or texts or classes, are just empty MenuItems. Bizarre. Either way, I found making a static cache copy of self.items() on the first pass fixes that problem. The Menu is visible of course on both calls, the UIA controls exists. But somehow pywinauto loses all the element info between these two calls to MenuItemWrapper.items().

Not understanding how or why frustrates me some and my solution here may not address the real issue as a consequence. But this does work and it has me rolling and I can access the menus.

Also puzzling to me is that once I click the menu. The menu has one child, itself. And so I click it, extract that one item, and get its children. That works!

[codecov]

Codecov Report

Merging #827 into master will decrease coverage by 0.03%.
The diff coverage is 0%.

@@            Coverage Diff            @@
##           master    #827      +/-   ##
=========================================
- Coverage   89.53%   89.5%   -0.04%     
=========================================
  Files          49      49              
  Lines       20638   20648      +10     
=========================================
+ Hits        18479   18481       +2     
- Misses       2159    2167       +8

[bernd-wechner]

Well, well, first Codacy moans about some white space at end of lines - FIXED and ammended commit.

Now AppVeyor finds a few failed tests. But I click Details and I see nothing even remotely lucid explaining what failed. Biggest fail for me is AppVeyor ;-). You think there'd be a nice way to see exactly which tests failed and, moreover, if they failed pre-PR or are only failing with the PR merged in? Lack of clarity in the extreme.

And codecov, well, of course I added a few lines to a function which, if not tested (covered) is now a weeny bit bigger and hence code coverage would drop a margin (0.04%). I guess what it's saying is that function should have a test!

[bernd-wechner]

It strikes me BTW, that perhaps given I filed #706 earlier this year, that you might not always want menu flutter in order to get the children, and there may be contexts in which they are available with out it. The latter is handled fine in the patch (menu only clicked if no children found), but it still makes the assumption that the MenuItem concerned is opening a submenu not a window. To wit a more general solution would add an optional argument to items(), a flag, which defaults off, and permits menu clicking to search for children.

So:

    _cache_items = {}
    def items(self, click_to_find_children=False):
        """Find all items of the menu item"""
        return self.children(control_type="MenuItem")
        items = self.children(control_type="MenuItem")

        # If a menu is not displayed then the UIA controls (children of self)
        # will not yet exist. We can bring them into existence by clicking the
        # menu (self). The controls can also (mysteriously) lose all their
        # element_info. So once we see them bv clicking the menu we cache them
        # for subsequent use.
        if not items and click_to_find_children:
            if self._cache_items.get(self.element_info.automation_id, None):
                items = self._cache_items[self.element_info.automation_id].copy()
            else:
                self.click_input()
                if self.children():
                    me = self.children()[0]
                    items = me.children(control_type="MenuItem")
                    self._cache_items[self.element_info.automation_id] = items.copy()

        return items

Am loathe to push a change into this PR without some feedback as to pulling it from the team. Waste of my time to keep tweaking this in unpopular directions. I do know, it's an enabler for the app I'm working with, and permits me to navigate menus easily.

[airelil]

I'm not by my computer, so just a short comment. The failing test_menu_by_mixed_match is a new failure, try to see why it stopped passing. The test_scroll and testCheckBoxes are known as failing

[vasily-v-ryabov]

Before looking into new failing test I would recommend thinking on the suggested approach first. Yes, test_scroll and testCheckBoxes are known flaky tests. Please ignore them. They have been fixed in atspi branch which is not merged into master yet. Hope to merge it till end of year.

[vasily-v-ryabov]

I wouldn't rely on automation_id property as a key of dictionary because it's possible to have child items without such property. I can't provide an example right now because of being too busy, but it's worth trying different apps with this approach to test everything works the same way or better.

[bernd-wechner]

Wow, time flies. I too am only looking at this again now. If automation_id is not reliable we could just check that it is available and if it is, then cache the item.

[vasily-v-ryabov]

Never say "Never!". :) Yeah, that may be better, but why not use self.element_info.name? Maybe it's a specific menu implementation. Is it Qt5 app?

[bernd-wechner]

Nope a VCL (Borland) app. I have of course since discovered how neat pywinauto's implementation of metaclasses is such that I simply define a new class derived from this one, and override items, and that is respected and used by pywinauto (the metaclass updates a dict mapping of controls to wrappers). Which means I'm actually free to experiment willy-nilly on broader impacts. Part of me thinks the earlier solution I suggest of an optional argument click_to_find_children=False is impact neutral on existing functionality and does nothing special.

[vasily-v-ryabov]

Thanks for the details! I will think what we can do.

[bernd-wechner]

Turns out rather that automation_id which indeed proved unreliably available, the only key I could find for a caching dict is element_info.element which seems to be a unique pointer (which Python permits as a dict key). Problem is nothing else I could find anywhere under element_info provides a unique handle on the element (include name which is blank on most of our controls).

[vasily-v-ryabov]

click_input() is an invasive technique which can make dump_tree() a.k.a. print_control_identifiers() blinking with the menu. Item texts are taken into account by best_match algorithm so dump_tree() can call it to obtain all possible best_match names. As I remember, I already fixed similar problem for combo box wrapper pretty recently (I mean last year :) ).

[bernd-wechner]

I'm not sure what you mean by "invasive" or what alternative there is.

[vasily-v-ryabov]

"Invasive" means that method .dump_tree() assumed as read-only. Think about it as a C++ const method. It calls .items() for every complicated element, but your implementation of .items() changes the state of the element (not a const method in terms of C++) and it even keeps submenu open. More right way to do it is expanding the submenu explicitly in your code, call what you need (list subitems) and then collapse the menu back if you don't need it any more.

Yes, some apps have better design of menu structure so the items are invisible but accessible. In your case a dynamic action is required to list the items.

[bernd-wechner]

As per original note above: Part of me thinks the earlier solution I suggest of an optional argument click_to_find_children=False is impact neutral on existing functionality and does nothing special. But I'm loving the easy wrapper override facility that makes experimenting with solutions easy.

[vasily-v-ryabov]

BTW, the commit was made with different e-mail not matched with your GitHub account. I always vote for correct account matching commits to count all the contributions properly. And thanks for the attempt!

Can we know which kind of application is used in your case? Is it publicly available?

[bernd-wechner]

BTW, the commit was made with different e-mail not matched with your GitHub account. I always vote for correct account matching commits to count all the contributions properly. And thanks for the attempt!

Can we know which kind of application is used in your case? Is it publicly available?

Sorry for the slow response here. Been nose down, and overlooked this. Not sure how and why the e-mail mismatch happens I have only one github account. Would need more details (clues) to approximate understanding what that is about.

As to the application it's a VCL front end: http://docwiki.embarcadero.com/RADStudio/XE6/en/VCL_Overview

it dates back to the '90s and is still a going concern, but that history and the VCL front end pose some challenges to automating UI interactions.

I'm rolling along though. The main obstacle is that most of the UIA elements don't have names such, and so I am having to find them using class_name and inference, and using dump_tree and using inspect.exe and spy++ prudently where needed (I sort of wish dump_tree was available from all controls not just windows - mainly because the app I'm working with has very few, typically just 1, window and a lot of Panes). Sadly it takes consistently 30 seconds after starting the app for top_window() to return and I have yet to diagnose why (that certainly seems excessive).