Experiments with extending testdriver.js for platform accessibility API testing

[spectranaut]

This works on linux for chrome, chromium and firefox:

./wpt run --force-renderer-accessibility --log-mach-level debug --log-mach - chrome core-aam/acacia/test-testdriver.html
./wpt run --binary <chromiumbinary> --force-renderer-accessibility --log-mach-level debug --log-mach - chromium core-aam/acacia/test-testdriver.html

./wpt run --log-mach-level debug --log-mach - firefox core-aam/acacia/test-testdriver.html

Issues:

1. The browser is found by "product name", so, looking for "chrome" or "chromium" or "firefox" in the application name -- so we can't have two of the same browsers running on a machine.
2. We can only one run test at a time. For linux: This is because we need to wait for the "document:load-complete" event for the specific page title. We need to pass back the title page with the name of the test, and keep track of what page we are testing (to know whether to wait for a new "document:load-complete" event.

Design questions:

1. Is this the right way to extend testdriver.js for this purpose?

[spectranaut]

@cookiecrook, thanks so much for your offer to take a look :) We are curious primarily, at this point, about the direction of extending testharness in this way. This executoracacia will be responsible for accessing different APIs based on the operating system. Obviously, its not related to webdriver, even though AcaciaPlatformAccessibilityProtocolPart is being "implemented" as a protocol part in the class WebDriverProtocol(Protocol) object.

You'll see a lot of files touched because I've bubbled up the "product name" in order to have it here, so we can find the correct browser to test the accessibility API of.

[cookiecrook]

I support any exploration for improving reliable testability of any aspect of web accessibility, including the platform APIs. I don't have strong opinions about extending the testharness this way. @gsnedders may have more ideas or questions.

That said, my primary questions came up in yesterday's call.

• It's not clear to me how platform-dependent API output will be tested in a platform-independent way using WPT.
  • As I understand it, each test expectation will be different per platform and, at least temporarily, would expose some implementation details.
  • Are the results for testing the Mac APIs intended to be listed as "N/A" for the Windows or Linux CI platforms?
  • Or is the intent to compare Mac-specific platform API results to Windows- or Linux-specific results? If so that'd be an Apples to Oranges comparison. Please forgive the terrible pun.
• I also wonder if this testing path might be logical to consider downstream of (after completing) AccessibleNode, rather than bypassing it.
  • What if there was a new accessor to get the platform-specific tree from the accessible node of an element? These may be similar to the WebKit or Chrome platform-specific tests.
    • for a DOM element (any element, including body), get the accessible node, then get the tree output (new accessor defined in webdriver?) for a specific platform parameter: [mac | atk | win_uia or win_ia2 | etc].
    • It might be similar to the serialized tree dump exploration the group just abandoned, but passing in a parameter to get a platform-dependent tree expectation instead of the non-standardized engine-internal representation of the tree.
    • Of note for others, these expectations are partially defined in Core-AAM and elsewhere. It would take a lot more effort to ensure those trees were interoperable. More toward this might come out of the tree walker exploration discussed in Interop Accessibility #90.

[alice]

Thanks for the comment! I'll reply with my thoughts, and @spectranaut can add hers later on.

It's not clear to me how platform-dependent API output will be tested in a platform-independent way using WPT.

• As I understand it, each test expectation will be different per platform and, at least temporarily, would expose some implementation details.

Yeah, we've talked about whether this project is a good fit for WPT, given WPT is generally testing browser internals and, as you point out, those are platform-independent by design philosophy; conversely, accessibility mappings are platform-dependent by necessity.

That said, I don't know that it's the case that it should expose implementation details. The intent for the scope is to test the Accessibility API Mappings specifications, such as Core-AAM. Obviously these don't cover the complete picture of what is involved in making web content accessible to assistive technology, but they should, in theory, be consistently implemented across browsers, shouldn't they?

• Are the results for testing the Mac APIs intended to be listed as "N/A" for the Windows or Linux CI platforms?

This is an important technical question: how do we implement platform-specific behaviour and expectations? I don't know enough about WPT to know what's currently technically possible, but I would hope that we could have platform-specific tests (since the APIs are different on each platform, they won't even be able to run on other platforms) which contain their own assertions about correct behaviour, and then have the appropriate tests collected and run by the runner based on the platform.

• that'd be an Apples to Oranges comparison. Please forgive the terrible pun.

This is magnificent.

• What if there was a new accessor to get the platform-specific tree from the accessible node of an element?

I like the approach prototyped here because it is a good integration test: because it goes via the same "read" API as the assistive technologies would, rather than inspecting the browser's internal representation of its accessibility tree.

That said, if this approach turns out to be infeasible, I think it would be good to expose platform-specific information through the webdriver API eventually (though we'd still run into the platform-specific expectations problem regardless.)

Also, I don't know that we need to get the tree - Valerie has implemented just getting the properties for a single Element, which allows us to test the AAMs, which was the goal of this work. Would restricting the API to the properties for a single Element resolve your concerns about exposing implementation details?

(It would also be good to have more standards about how the tree is built, and as you know I'm excited about that work! Thanks for the pointer to the Tree Walker work, I didn't know about that until now.)

[spectranaut]

Thanks James for all the thoughts, it brings up a lot to think about, and thanks Alice for the replies, I don't actually have a lot to add :)

I have vaguely thought of handling the platform specific tests in one of two ways, neither is perfect:

1. We have a test files that are platform specific, and the test harness only runs the tests relevant to the platform. So, when on mac, only run the mac test files. I'm not sure how this will look in wpt.fyi, it would require some work and exploration -- but at the file level we would see some files don't apply to some platforms: https://wpt.fyi/results/wai-aria/role?label=experimental&label=master&aligned

2. We could also write a test that contains expectations for all platforms, request the platform accessibility node, then the based on the response (did I get a AX API node, or a Atspi node) processes either AX API expectations or Atspi expections as appropriate. This would also look weird in wpt.fyi, because the expectation is in the name, so would we list all platform expectations in the name (for example)? Also there would be a different number of expectations depending on the platform.

[alice]

I believe there's an open question (still @cookiecrook?) with the WebDriver accessibility APIs about whether to have a way to get this kind of bag of properties, or to instead behave more like the rest of the WebDriver APIs and only support fetching single properties.

We should probably consider for ourselves which will work better for us as well - getting the bag of properties might be slow if any of the properties are slow to compute or cause exceptions, etc.; however, accessing individual properties instead would probably require a kind of "stringly typed" (i.e. passing a string representing the property to query) API rather than having every single property we might want to test on each operating system as its own method.

Still, it might be worth prototyping what it might look like to access single properties as well as this bag of properties API?

[alice]

Thanks for the comment! I'll reply with my thoughts, and @spectranaut can add hers later on.

It's not clear to me how platform-dependent API output will be tested in a platform-independent way using WPT.

• As I understand it, each test expectation will be different per platform and, at least temporarily, would expose some implementation details.

Yeah, we've talked about whether this project is a good fit for WPT, given WPT is generally testing browser internals and, as you point out, those are platform-independent by design philosophy; conversely, accessibility mappings are platform-dependent by necessity.

That said, I don't know that it's the case that it should expose implementation details. The intent for the scope is to test the Accessibility API Mappings specifications, such as Core-AAM. Obviously these don't cover the complete picture of what is involved in making web content accessible to assistive technology, but they should, in theory, be consistently implemented across browsers, shouldn't they?

• Are the results for testing the Mac APIs intended to be listed as "N/A" for the Windows or Linux CI platforms?

This is an important technical question: how do we implement platform-specific behaviour and expectations? I don't know enough about WPT to know what's currently technically possible, but I would hope that we could have platform-specific tests (since the APIs are different on each platform, they won't even be able to run on other platforms) which contain their own assertions about correct behaviour, and then have the appropriate tests collected and run by the runner based on the platform.

• that'd be an Apples to Oranges comparison. Please forgive the terrible pun.

This is magnificent.

• What if there was a new accessor to get the platform-specific tree from the accessible node of an element?

I like the approach prototyped here because it is a good integration test: because it goes via the same "read" API as the assistive technologies would, rather than inspecting the browser's internal representation of its accessibility tree.

That said, if this approach turns out to be infeasible, I think it would be good to expose platform-specific information through the webdriver API eventually (though we'd still run into the platform-specific expectations problem regardless.)

Also, I don't know that we need to get the tree - Valerie has implemented just getting the properties for a single Element, which allows us to test the AAMs, which was the goal of this work. Would restricting the API to the properties for a single Element resolve your concerns about exposing implementation details?

(It would also be good to have more standards about how the tree is built, and as you know I'm excited about that work! Thanks for the pointer to the Tree Walker work, I didn't know about that until now.)

[alice]

We probably want to have eventually separate executor implementations per platform, and then a cross-platform implementation which can import the appropriate one and call its get_accessibility_api_node (or whatever API we go with) method.