Usability with screen readers

[saulshanabrook]

I am opening this issue to track being able to use JupyterLab easily with a screen reader. @SamKacer on gitter raised the issue that they are not able to edit files in JupyterLab with a screen reader.

We use the CodeMirror editor for the built in file editor and the cell editors, which is not usable with screen readers (jupyter/notebook#1801 (comment) codemirror/codemirror5#4604).

There is an extension that adds a Monaco based text editor (https://github.com/jupyterlab/jupyterlab-monaco) and Monaco looks like it has support for screen readers, so using that extension might provide users so better support.

It might also make sense to add tests for screen reader based navigation. I am not sure if this is doable with selenium or something.

[SamKacer]

@saulshanabrook Thanks for informing me about the other editor. I will test it out first and report on its accessibility. If it is good enough I think that is sufficient for screen reader users. If not, I will expand on SR accessibility.

[SamKacer]

Ok, so I tried using the Monaco editor extension, but couldn’tthere seems to be an issue with installation, so I am waiting to get a reply on that. As I understand the Monaco editor is the editor in VS code. I use VS code for its editor, so getting it to jupyter lab would be great if it was equally accessible.

In the meantime I will expand on the issue with the default editor and explain the little I know about using ARIA markup for designing accessible web apps that might be relevant in improving the accessibility of the default editor.

Firstly, I will give a brief explanation of how screen reader users review and navigate text. So, when I am in a editing field, I get feedback on the chars I tuype and the words as a whole after I press spacebar when I am done writing it. Very important is the reviewing of text, since I cant see it I have to have the screen reader read back any text I want to review.. there is a command I can envoke to report the current line of text. Also when I navigate around the text with the editor caret (cursor) the SR will report the text as I am navigating through it thusly:
If I press up or down, going up or down a line, the SR reads the line I have navigated to. If I just move by char, by pressing left or right, the SR reads the char under the caret. If I navigate by word (ctrl+left/right) the SR reads the word under the caret.

The open source SR I am using, NVDA, which is one of the most popular SRs also has a review cursor, which basically allows the user to review text in a similar way as moving around using the editor cursor (system caret) using the num pad, but gives the advantage of not moving the system caret, so the user can quickly review some text maybe even copy some and then continue typing exactly where they left off, because the system caret hasn’t moved.

If anyone wants to have a quick feel for how a SR user experiences text editing, you can download NVDA free and test it on your machine. You can still use the mouse and keyboard as normal, but you get an extra layer of speech feedback and keyboard commands. Learning the keyboard commands isn’t vital for a sighted user who just wants to know what it is like when you edit text, since you can just navigate around with the mouse and alt+tab as usual, but I will just mention the command for toggling speech on/off if it gets annoying while you are trying to just navigate around:
NVDA key + S
(NVDA key is either capslock or insert)

For linux users there is orca which is built in at least for ubuntu, not sure about other distros. And MacOS has voice over built in, but it has slightly different reporting of text in editors, but same in principle.

A editor that supports all the aforementioned features is notepad or Word, wordpad, etc.
Most web app editing fields (i.e. in forums) are also fully accessible, so if you’d like to get a quick feel just open up notepad and have a try.

So, here I will now discuss how the default editor could be made more accessible. I will first discuss how it could be made 100% accessible having all the mentioned text review features and progressively propose versions with less accessibility features, but less change to the structure and behaviour of the editor, since implementing perfect SR accessibility might not be practical.

So, just navigating through the webapp with my SR, it seems the editor for the cell is structurally split up into a multi line editing field widget and a separate group of elements for a actually displaying the contents.

To allow for all the SR features mentioned to work, all of the text must be in the html edit field widget. Presently, , when the text typed, it disappears from the edit field, so the SR cant access it anymore while editing. If the whole file remained in the edit field, the SR user could review the text using both the system caret and the SR’s virtual review cursor.

Alternatively, if at least the current line of text was inside the edit field widget, the SR user could at least review the current line and if when the user switches between lines (up arrow/down arrow) then they could review the text as they normally would by moving around with the system caret.

Finally, if the structure of Code Mirror simply wouldn’t allow for this, then a final workaround which wouldn’t produce great accessibility, but the editor would be workable, would be to use a trick to send messages to the SR by implementing a little ARIA markup trick.

You could have a html element that would update with info about the text so the SR user could review it easier. So, for example, if you’d move the editor cursor down a line, this element would change its text contents to that line. The trick is that this html element would have the ARIA-live tag, which signals to the SR whenever the contents of this html element change, then read those contents to the user. This html element could also be made invisible with another ARIA tag, which only makes it visible to SRs.

So, effectively a webapp can send messages for an SR to read, without it changing the experience for a non-SR user.

Anyway, these are my initial thoughts, I can expand on anything if this is resonating with any devs.

[SamKacer]

going back and playing around with google docs, it looks like it handles accessibility with the lattr approach with ARIA live regions.

here is a document which details using ARIA as well as web accessibility in general:
https://www.w3.org/TR/aria-in-html/

Also, didnt mention, but the menu bar is usable, but only using a SR workaround. The best accessibility of elements like menu items is for them to be focusable using keyboard actions (like tabbing) and activateble by either inheriting from already activatable elements like html native buttons or links, or by scripting the activatability. This allows the SR user to activate the menu item or whatever interactive element by pressing space when the system focus is on the item.

It is also useful to telegraph the role of the elements if they are not native html widgets such as in case of the menu bar items in jupyter lab, by applying the role attribute with the right value to the div containing the element.

currently, the menu bar is usable using a SR workaround. I can find the menubar using the system caret navigation and use a SR function to move the computer mouse pointer to it, then click the mouse to activate the menu bar item. this works, but feels kind of dirty to an SR user and isnt very efficient for workflow. Implementing improved acdcessibility in the menu bar is overall much simpler than the editor issue.

[ian-r-rose]

Thanks a ton for your input @SamKacer. Improving the accessibility of JupyterLab (especially with screen readers) is important to us, though our progress on that front has been slow, largely due to lack of expertise on our end.

A couple of months ago we had an accessibility audit with a SR with the UC Berkeley DSP office. JupyterLab is being deployed to students at a large scale at Berkeley, and accessibility is a priority. The conclusions of that meeting were consistent with what you are saying here. In particular, the CodeMirror editor is largely unusable with a SR.

You are right that Monaco is the same underlying editor used in VS Code, and so switching to that would be a great improvement. In fact, that is one of the primary reasons that we want to move to Monaco.

Our intention is for the menus to be more accessible than they apparently are right now. Once they are active I can navigate around them with the arrow keys, but I see now that they are not tab-activatable as it is (ping @sccolbert). Would it be a good place to start to (1) give the top-level menus a tab-index, and (2) add an aria role attribute to them?

Again, thank you very much for your feedback. As we move forwards with this process, we can add more information to this issue (and would welcome further thoughts!).

[SamKacer]

@ian-r-rose
No problem. I am pretty excited about the jupyter project from what I hear from my sighted counterparts, but right now as a blind user I cant fully enjoy it. I am interested in exploring data science and jupyter seems to be an excellent platform for such things. Increasingly more visually impaired people are getting into the IT sector and having jupyter lab be accessible would be a massive boon for the VI community, so I am very happy to that accessibility is important to you.

If the VS code editor could be used in jupyter lab that would be great if it retained all its features and accessibility it has in the desktop version. Looking at the front end for VS code, it looks like the GUI is basically made of html, so hopefully if it was incorporated into jupyter lab, its functionality would remain the same.

Yes, the changes you propose are steps in the right direction, but more would need to be done for the menubar to be considered fully accessible. I need to stress that the ARIA role attr is merely semantic. It allows the SR to inform the user about its semantic usage and so they expect certain keyboard control featuresto be implemented (i.e. pressing left or right arrow will move between the items on the menu bar, pressing down, space, or enter while focused on an item will open the submenu and move the focus to its first item, etc.).

Right now while researching ARIA, I found these two great documents which not only describe using ARIA, but designing accessible webapps in general.

This one I already linked to, but is a great starting point:
https://www.w3.org/TR/aria-in-html/

and here is a doc where among other things it describes the keyboard control conventions for the ARIA roles and example implementations including specifically the menu bar:
https://www.w3.org/TR/wai-aria-practices/#menu

the above link directly goes to the section about menu bars.

For webapps to be SR accessible, having comprehensive keyboard navigation in place is key. ARIA role attrs help describe to the SR user what sort of keyboard interaction convention to expect, but don’t actually implement it. Fortunately, applying ARIA roles and states is very simple, but thankfully keyboard nav which is harder to implement is useful for all users, not just for SR users.

So, as far as the menu bar the latter link comprehensively explains and demonstrates all the changes that would make it 100
% accessible.

Also I will mention an important point, when referring to focus, for focus switching to be useful to a SR it needs to be the system focus and not a virtual focus. Not sure how keyboard nav was planned on being implemented, but I am mentioning this just in case.

anyway, I will be happy to test the changes as they are implemented, but the doc is pretty straightforward as to what needs to be implemented and the keyboard nav can be tested by any user, while ARIA roles, states, etc. are very simple to apply.

[SamKacer]

Hello all,

I have thought of a quick fix, which might be pretty simple to implement and would allow for accessibility of the editors, which is the primary screen reader accessibility problem.

What if the user could toggle a “simple editor mode”, which would replace the code mirror editors with plain html multi line edit fields? I know the advanced editor features such as autocompletion wouldn’t be available, but at the current stage, the only accessible feature in CM is copying the contents over to an external accessible editor anyway.

Would this be possible?

[jasongrout]

Or it may be simpler to have two notebook renderers. You right-click and "open with" the simplified notebook editor (or you can set it to be the default).

[vidartf]

@SamKacer The editor is already an extensible point, so making a third-party extension that implements a minimal editor would be an interesting exercise (not one that I can volunteer for though). See the IEditorServices interface in packages/codeeditor for the functionality that would need to be implemented.

[jmh-aira]

Hi, I wanted to follow up with th is? has any headway been made in making Jupyter accessible with an SR?
Thanks.

[tonyfast]

crossref #9399

[isabela-pf]

Since we're returning to this issue with some further discussion at #9399, I'm taking @SamKacer's advice to follow up on this issue so anyone following this but not newer issues gets updated too.

Three potential solutions proposed by @jasongrout

• CodeMirror 6 (at that time just starting) looked like it would be significantly better for accessibility - I still would love if someone could try out the CodeMirror 6 demo at https://codemirror.net/6/ to test this assumption
• Monaco - we have an experimental plugin changing the text editor for monaco, but it suffered from other issues, including for a long time monaco wasn't really packaged well for consumption outside of VS Code, and that it was pretty RAM intensive compared to codemirror, which would possibly make long notebooks suffer a lot in our current architecture.
• We could fall back to just a straight HTML text area, which certainly is accessible, even if it does not offer any coding help.

This issue (and #9399's discussion) were brought up at this week's JupyterLab general team meeting. This isn't a meeting where we came up with solutions, but gave a heads up to more people that there's interest in proposing changes to the code editor, a major part of JupyterLab. Quoting my own summary of the meeting:

• There is not an agreement on which solution to pursue, but I expected that at this point. I personally think exploring the options is one of the next steps.
• Out of who attended this week, there wasn’t any outright opposition to proposing changes to the code editor for this reason (yay!).
• The big takeaway is that this is not a 3.0 change but a 3.x change at the earliest, possibly 4.0. Some people in this meeting did agree this should be one of the first priorities for the next release and that it should get triage tags and appropriate attention as soon as 3.0 is out.

I also want to note that I have this on the agenda to discuss at the JupyterLab accessibility meeting next Wednesday (Dec 16 at 10:15 AM Pacific; on the Jupyter Zoom channel), where I hope to talk more about where this fits on our priority list and the proposed solutions so we are prepared to bring work back to the rest of the team sooner than later. Anyone is welcome to join this meeting and/or other the other accessibility meetings that are happening (every other Wednesday at 10:15 AM Pacific on the same channel). We are working to address the backlog of accessibility issues that we've left untouched for (in my opinion) too long, and these meetings are meant to coordinate those efforts.

[fcollonval]

I follow from afar the discussion. And I think there is a important point that has not been mentioned about the editor: the compatibility with non-desktop browsers. This is not per se related to accessibility. But monaco editor is not for mobile browser (as clearly stated in the project homepage) when CodeMirror is. It may be a blocking point when choosing a more accessible editor.