Define meaning of assertion priorities: "MUST", "SHOULD", and "MAY"

[mcking65]

We implemented ARIA-AT-APP Issue 737 - Revise reports to show assertion priorities as MUST/SHOULD/MAY instead of REQUIRED/OPTIONAL but have not yet documented how either assertions or assertion/command pairs should be mapped to each priority level and how readers of aria-at reports should interpret the priorities.

In this issue, we will align on consensus definitions for each priority with clear guidance on how to assign priorities and interpret reports and update the ARIA-AT Glossary definition of Assertion Priorities.

[mcking65]

Edited March 6, 2024 to incorporate feedback from Feb 29 meeting.

Proposal for new definition ...

Specifies the requirement level of an assertion (MUST, SHOULD, or MAY).

The ARIA-AT assertion priority indicates which one of the following three levels of requirements is expressed by an assertion. While ARIA-AT tests are not directly associated with a normative specification, the requirement levels are modeled after Key words defined by RFC 2119 for indicating requirement levels. This alignment could thus be used as the basis for developing a specification in the future.

1. MUST: The assertion is an absolute requirement. The command or event being tested MUST result in the behavior described by the assertion. Failure to do so could block users of the command from perceiving, understanding, or operating the type of content represented by the test case.
2. SHOULD: The assertion is a strongly recommended requirement. The command or event being tested SHOULD result in the behavior described by the assertion. Failure to do so is likely to impede users of the command from perceiving, understanding, or operating the type of content represented by the test case. There may exist valid reasons in particular circumstances to ignore this assertion, but the full implications must be understood and carefully weighed before choosing a different course.
3. MAY: The assertion is truly optional. The command or event being tested MAY result in the behavior described by the assertion. Failure to do so is not likely to impede users of the command from perceiving, understanding, or operating the type of content represented by the test case. One assistive technology (AT) vendor may choose to provide the asserted behavior because a particular marketplace requires it or because the vendor feels that it enhances the product while another vendor may omit the same behavior.

NOTE: Assessment of impact on users is generally based on the following assumptions:

1. The AT user does not possess a specific skill level, i.e., may be a novice user of the AT.
2. Default AT configuration is being used unless otherwise specified by the test.
3. AT may provide configuration options that enable users to change any of the behaviors asserted by a test.

[css-meeting-bot]

The ARIA-AT Community Group just discussed Assertion priority definitions.

The full IRC log of that discussion

<jugglinmike> Topic: Assertion priority definitions
<jugglinmike> github: https://github.com//issues/1033
<jugglinmike> Matt_King: People have been asking, "what do MUST, SHOULD, and MAY mean?"
<jugglinmike> Matt_King: I read RFC2119 which defines those terms for normative specs, and I came up with definitions for ARIA-AT
<jugglinmike> Matt_King: So I'm looking for feedback
<jugglinmike> Matt_King: In the first comment of this GitHub issue, you'll find my proposal for the definitions to place in the glossary
<jugglinmike> Matt_King: The first question: ignoring for now whether you agree with what's written here, do people understand what is written here?
<jugglinmike> James_Scholes: I don't understand the final sentence of the "MAY" definition.
<jugglinmike> James_Scholes: that is, "An AT which does not provide the asserted behavior MUST be prepared to interoperate with another AT that does provide the behavior, though perhaps with reduced functionality."
<jugglinmike> James_Scholes: Do you have a practical example?
<jugglinmike> Matt_King: I borrowed that language directly, just inserting the word "behavior"
<jugglinmike> Matt_King: I was thinking about an ARIA feature like something related to digital publishing or "annotations" in the ARIA working group...
<jugglinmike> Matt_King: ... Screen readers provide a lot of different kinds of presentations of that sort of material
<jugglinmike> Matt_King: I'm imagining that there might be something asserted that happens by default
<jugglinmike> Matt_King: "SHOULD" is that "we really think you should do this" but with "MAY", it's that "we think you could interoperate regardless of whether this is done"
<jugglinmike> Matt_King: When it says "must be prepared to interoperate", then the AT which doesn't exhibit the described behavior, it should exhibit some other equivalent
<jugglinmike> Matt_King: But we might want to leave that statement out
<jugglinmike> James_Scholes: I expect that this is likely to come up often (particularly because it uses the word MUST), and because it requires so much explanation, it would be better to leave it out
<jugglinmike> Matt_King: Let's say we hypothetically thought it was a good idea that screen readers automatically announce all assertions. We would say that they "may" do this. If a screen reader didn't do this automatically, but did do it as they entered elements, that would be satisfied...
<jugglinmike> Matt_King: ...unlike a third hypothetical screen reader which never announced insertions
<jugglinmike> s/all assertions/all insertions/
<jugglinmike> James_Scholes: My intuition is telling me to leave that off
<jugglinmike> Lola: So because this is in a spec, we're talking about tests. When it comes to "MAY", it's possible that VoiceOver does something that is a "MAY", and that JAWS doesn't do that thing. To call JAWS's behavior "acceptable" sounds weird to me from an interoperability standpoint
<jugglinmike> Matt_King: We'll discuss the subsequent agenda item here in this context
<jugglinmike> Matt_King: Vispero does not think the word "alert" should be spoken when an alert is announced
<jugglinmike> Matt_King: We've already demoted the relevant assertion to a "should" based on this feedback. They'd like it further demoted to a "MAY" assertion
<jugglinmike> Matt_King: Given that much information and these definitions alone, what do others think?
<jugglinmike> Hadi: There are a lot of visual elements which speak for themselves.
<jugglinmike> Hadi: When we, as screen reader users, we don't have the visual cues. I am for the idea to have the role of "alert" spoken aloud
<jugglinmike> Matt_King: That's a perfect explanation for the original motivation back in 2010
<jugglinmike> Matt_King: Here's what's happened since 2010. Many chat apps right now use alert. On every single message, you hear "alert" as a prefix constantly
<jugglinmike> Matt_King: As a result, it no longer serves the "attention grabbing" purpose it was originally designed to serve
<jugglinmike> Hadi: Perhaps within a web application. But when an alert comes up in the context of another application, that's a different story
<jugglinmike> Matt_King: Agreed. Vispero can't fix that, but Microsoft is working on a technical solution to this. It will be available in a year or two
<jugglinmike> Matt_King: Vispero's argument right now is that in the current state of the world, "alert" is kind of meaningless
<jugglinmike> Lola: I am inclined to push back a bit. I'm not married to this perspective, but it sounds like we are adapting our tooling to match tooling which is not accessible in the first instance
<jugglinmike> Lola: If a web developer misuses alert, that's not accessible. It seems strange to me to change our tests to match that
<jugglinmike> Matt_King: I very much agree. That's where we fall almost all of the time. It's just that this is a strange case: if you're a developer, you have no other choice
<jugglinmike> Matt_King: This is why the APG has not written guidance for live regions
<jugglinmike> Matt_King: If you want live regions to work across browsers and ATs, you have to use alert
<jugglinmike> Matt_King: If you're targeting only certain user-agents, then sometimes you can make live regions work without alert
<jugglinmike> James_Scholes: I am sympathetic to Vispero's comments, but it's kind of a multi-pronged issue. As a user, their stance makes sense to me. As an accessibility professional, this is less the case.
<jugglinmike> James_Scholes: "aria live" still doesn't work after 14 years. At some point, you have to say "enough is enough". The same goes for "role=status".
<jugglinmike> James_Scholes: If something has not worked well for a decade, my sympathy for the people who should be implementing somewhat dips
<jugglinmike> Matt_King: But this isn't something Vispero can fix. The problem is on the browser side
<jugglinmike> James_Scholes: Good point
<jugglinmike> James_Scholes: But of course, Vispero's feedback is valid and based on authentic real-world experience
<jugglinmike> James_Scholes: I'm really torn, to be honest. One of the points that's made here in this issue is about users possessing certain skill levels. I really think that's an issue here. For a novice user, knowing the difference between an alert and a focus change, I think is key.
<jugglinmike> Matt_King: We're out of time for the day, but we didn't get to hear from everybody.
<jugglinmike> Matt_King: I encourage folks here to comment on the definitions (aria-at #1033), and you can also comment on this specific example there
<jugglinmike> Zakim, end the meeting

[jugglinmike]

Thanks for following up on this, @mcking65. As I understand the normative context, ATs MAY convey information beyond what they MUST and SHOULD convey with the caveat is that the CG reserves the right to judge certain responses as "excessively verbose." This makes me think that explicit "MAY" assertions are meant as an assurance to implementers--an assurance that the CG will not judge certain responses to be "excessive." Is that right? Do they serve any other purpose?

[css-meeting-bot]

The ARIA-AT Community Group just discussed Define meaning of assertion priorities: "MUST", "SHOULD", and "MAY".

The full IRC log of that discussion

<jugglinmike> subtopic: Define meaning of assertion priorities: "MUST", "SHOULD", and "MAY"
<jugglinmike> github: https://github.com//issues/1033
<jugglinmike> Matt_King: We have "required" and "optional" defined in the glossary, but that's now out of date. This issue proposes definitions for "MUST" "SHOULD" and "MAY" that we can add to the glossary
<jugglinmike> Matt_King: We're able to consider these in terms of feedback we've recently received from Vispero about some specific expectations in the "alert" test plan
<jugglinmike> Matt_King: Similarly, we've received feedback from Apple about the "not pressed" state for toggle buttons
<jugglinmike> Matt_King: Last week, we were just talking about "SHOULD" and "MAY"; it didn't seem like there was any feedback on the "MUST"
<jugglinmike> Matt_King: I do want to consider "MUST" in this meeting, though, because I'd like to arrive on a final decision on the meaning
<jugglinmike> "The assertion is an absolute requirement. The command or event being tested MUST result in the behavior described by the assertion. Failure to do so could block users of the command from perceiving, understanding, or operating the type of content represented by the test case."
<jugglinmike> Hadi: I completely support that
<jugglinmike> Alyssa: Yeah; there isn't much room for interpretation there
<jugglinmike> Matt_King: I'm trying to take the definition of MUST from RFC2119 and translate it to an assertion in a test (e.g. "the role alert is conveyed")
<jugglinmike> Matt_King: I'm trying to explain the consequence of any one of our thousands of assertions
<jugglinmike> Alyssa: It makes sense to me. It differentiates between something like "SHOULD"
<jugglinmike> Hadi: English is my third language. I think "likely" maybe doesn't go very well with "MUST"
<jugglinmike> Matt_King: Well, we're not certain that it will block users because we can't be sure that something will block them
<jugglinmike> Matt_King: We don't know for sure if one single failed assertion would block somebody. It could, but it might take multiple failures
<jugglinmike> Hadi: I am for equal access. For instance, if you tab to an element and you get information that's different than if you arrive at it some other way. That's unecessary cognitive load
<jugglinmike> Matt_King: That sounds right to me. We have an assertion for "the role 'toggle button' is conveyed". If an AT conveyed only "button", you could still use that button--it wouldn't block you
<jugglinmike> Hadi: If I am uncertain at any stage, I call that a failure
<jugglinmike> Matt_King: Agreed. We're calling this a failure. We're just going on to explain what a "failure" means
<jugglinmike> IsaDC: I have a similar opinion, though English is also not my native language
<jugglinmike> Matt_King: The actual definition is just the first sentence. Everything after that is just an explanation
<jugglinmike> Hadi: Ah, in that case, I'm comfortable with this
<jugglinmike> Matt_King: Moving on to the definition of "SHOULD"...
<jugglinmike> "The assertion is a strongly recommended requirement. The command or event being tested SHOULD result in the behavior described by the assertion. Failure to do so is likely to impede users of the command from perceiving, understanding, or operating the type of content represented by the test case. [...]"
<jugglinmike> "[...] There may exist valid reasons in particular circumstances to ignore this assertion, but the full implications must be understood and carefully weighed before choosing a different course."
<jugglinmike> Matt_King: Again, modeled after RFC2119
<jugglinmike> Alyssa: Makes sense to me!
<jugglinmike> Matt_King: Then, for "MAY"
<jugglinmike> "The assertion is truly optional. The command or event being tested MAY result in the behavior described by the assertion. Failure to do so is not likely to impede users of the command from perceiving, understanding, or operating the type of content represented by the test case. [...]"
<jugglinmike> "[...] One assistive technology (AT) vendor may choose to provide the asserted behavior because a particular marketplace requires it or because the vendor feels that it enhances the product while another vendor may omit the same behavior."
<jugglinmike> Hadi: Can you elaborate a little bit on when vendors might choose?
<jugglinmike> Matt_King: Some screen reader developers have different ideas about what will better-serve their users
<jugglinmike> Matt_King: Some screen reader vendors might prioritize a certain market, e.g. people who aren't blind but are using a screen reader for some other purpose
<jugglinmike> Matt_King: They may make decision decisions based on these differing goals
<jugglinmike> s/make decision decisions/make different decisions/
<jugglinmike> Hadi: Thank you. The default setting is based on which level of verbosity?
<jugglinmike> Matt_King: Whatever the AT provides immediately after installation. In other words, we use the setting that the vendors themselves implement as default
<jugglinmike> Hadi: I'm concerned that AT vendors will map the "MUST", "SHOULD", and "MAY" to their verbosity levels
<jugglinmike> Matt_King: I doubt it. I don't think it's that neat and simple. Especially when it comes to "SHOULD". It's hard to imagine that they would stop doing those things on a medium verbosity
<jugglinmike> Matt_King: For instance, announcing the position within a radio group (e.g. "one out of three" or "two out of three"). We mark that as a "SHOULD" because you can technically get by without that
<jugglinmike> howard-e: The word "omit" in the definition of "MAY". To me, in English, that only sometimes represents intentionality
<jugglinmike> howard-e: Would it be safer to say something like, "while another vendor may not support the same behavior by default" ?
<jugglinmike> Matt_King: We actually do want that sense of intentionality
<jugglinmike> Matt_King: Sometimes, users of one AT actively don't want certain behaviors, so the implementers want to leave it out on purpose
<Sam_Shaw> jugglinmike: We are both saying you are able to do what you want within reason, and also you shouldn't do these things. My Questions is, if you do one of these things that we are saying they may do, we will consider it excess verbosity
<Sam_Shaw> Matt_King: With Voiceover or NVDA there is a message it gives, the help message, which we don't test, is wrong. If it is wrong, I think its legitmate to call it out as a problem
<Sam_Shaw> Matt_King: I wouldn't call it excess verbosity, I would call it confusing verbosity
<Sam_Shaw> Matt_King: The other reason for this, is if one SR developer does something and another doesn't, if you are unfamiliar with SR and your testing your component with a SR that does this behavioir, then test with another SR that doesn't, you would think its a bug in your code, and if not then you would think its okay that this behavoir does or doesn'
<Sam_Shaw> Matt_King: doesn't happen
<Sam_Shaw> jugglinmike: I want to guard against confusion when we say you can do anything you want, and you may do this. For me tying into the excessive verbosity reasoning helps clarify that
<Sam_Shaw> Matt_King: Lets push further discussion of Alert role to next week

[mcking65]

The glossary definition of assertion priority is now updated.