Update Sphinx to work with docutils-0.18.x

[ViktorHaag]

Is your feature request related to a problem? Please describe.
Docutils 0.18.0 has been released and at first glance it appears to have changes in it that are incompatible with Sphinx (cloning the Sphinx repo, running the tests clean; then installing docutils 0.18, updating Sphinx to allow docutils-0.18 and include it in the test suite, shows a whole bunch of failed tests).

Describe the solution you'd like
Make the changes to Sphinx required to be compatible with docutils 0.18.x.

Describe alternatives you've considered
The alternative would be to remain pinned to 0.17.x (ie "docutils packages with a version less than 0.18"); this may cause disruptions in projects and platforms that assume they can freely update dependencies to the latest versions, especially if they directly depend on docutils.

[astrojuanlu]

Some changes have been made already to support docutils 0.18, see for example #9701, #9724, #9725. However, it's still pinned to <0.18:

sphinx/setup.py

Line 26 in 2be9d6b

'docutils>=0.14,<0.18',

[nicobrevin]

Just in case anyone else is on an old version of sphinx, I had to pin my docutils version like so:

 sphinx==1.7.6
+docutils==0.17.1
 doc8==0.5.0
 sphinx_rtd_theme==0.4.1
 m2r==0.1.15

Without it I was getting this error:

Exception occurred:
  File "/usr/local/lib/python2.7/dist-packages/sphinx/util/nodes.py", line 57, in apply_source_workaround
    for classifier in reversed(node.parent.traverse(nodes.classifier)):
TypeError: argument to reversed() must be a sequence

The version constraint is pretty loose in that version of sphinx:

Collecting docutils>=0.11 (from sphinx==1.7.6->-r /tmp/requirements.txt (line 1))

Hopefully this will help another weary traveller searching for a solution after their build suddenly failed...

[tk0miya]

Now all tests are passed with the HEAD of docutils (see https://github.com/sphinx-doc/sphinx/actions/workflows/docutils-latest.yml).
It's time to support docutils-0.18. I'll merge #10164 if manual testing with docutils-0.18 will be also passed.

@eric-wieser @astrojuanlu Could you make a plan for testing, please?

[eric-wieser]

@tk0miya: are you tagging me because I was involved in discussion about a docutils upgrade that I've forgotten (which is entirely possible), or just because I've helped with Sphinx before?

[tk0miya]

@eric-wieser I'm very sorry. I'd like to notify another Eric (@ericholscher). Thank you for letting me know my mistake.

[ericholscher]

/cc @agjohnson who is probably a better person for design-related things. I know there was also some discussion around testing in #10070, but just getting back from vacation and catching up on these conversations.

[tk0miya]

We have a plan to release 5.0.0 in May. So I'd like to move forward. What is a barrier of this?

[AA-Turner]

The only remaining barrier as far as I can tell is that themes will need to update their CSS for changes to the Docutils HTML5 writer.

Docutils changed the following things from 0.17 to 0.18:

• Changed from <div> to <aside> for footnote text and citations, topics (except abstract and toc), admonitions, and system messages.
• Changed from <div> to <nav> for the Table of Contents.
• Uses child selectors and :first-child, :last-child pseudo classes instead of the compound-first, compound-middle, or compound-last classes for elements nested in a compound.
• Changed the name of the fn-backref CSS class to backrefs for <span>s of back-references.

To me this doesn't seem a blocker for Sphinx itself changing the pin, although perhaps we should ensure major themes are at least aware of what they'll need to do to support Sphinx 5 by raising an issue on their trackers?

A

[tk0miya]

Is anybody working on this? It seems stuck. As I commented on introducing the pinning strategy, unpinning is costly. I'd like to move this forward if nobody works on this.

[ericholscher]

@tk0miya I think that updating the pin in Sphinx to the latest released version of docutils is good. Keeping things pinned is important to ensure it doesn't break in the future, but I think the latest Sphinx should definitely pin the latest docutils version. It will likely still cause issues with some HTML output in various themes & configurations, but I think we've had plenty of warning before this change 👍

[collares]

With docutils 0.18.1, Sphinx triggers the following deprecation warning:

FutureWarning: The configuration setting "embed_images" will be removed in Docutils 1.2. Use "image_loading: link".

See also #9701 (comment)

[stsewd]

Not sure if this is intended, but the footnotes in the sphinx13 theme look like this with docutils 0.18.1

0.18.1

0.17

[tk0miya]

Okay, let's upgrade docutils to 0.18. I'd like to say thank you to all contributors!