Bump Docutils dependency

[AA-Turner]

Feature or Bugfix

• Dependency

Purpose

closes #9777 by bumping the declared support for Docutils to 0.18

A

[tk0miya]

Before moving the pin, we need to test it with docutils-0.18 at first.
#9091
#9056 (comment)

[AA-Turner]

Before moving the pin, we need to test it with docutils-0.18 at first.

I've read the two comments you linked and I'm not sure I understand what you mean -- is this beyond the normal test suite? (Said normal test suite is passing for this PR on Docutils 0.18).

Happy to do or help with any further testing that might be needed, of course.

A

[tk0miya]

Our test suite only checks the structure of HTML output partially. But it does not check how it is rendered. So we need to check it manually. Please ask the RtD team what the manual test is.

[astrojuanlu]

Please ask the RtD team what the manual test is.

cc @ericholscher @humitos @nienn

[AA-Turner]

But it does not check how it is rendered. So we need to check it manually.

if the html & css are exactly the same I imagine this would have the same rendering. Visual differencing tools exist, although I don't know any off the top of my head for CI. It might be useful to agree something that would make these upgdrades easier in the future than generic manual testing. (No offence intended to the manual testers, only that automated tools are less prone to forgetting / human error, and survive personnel changes etc).

If somebody lets me know what the manual testing process is I'm happy to support and potentially look into automating this for the future?

A

[ericholscher]

We'd love to have more automated tests. The main manual test is just building the docs with the RTD theme and latest docutils, and comparing the output, I believe. @agjohnson might have more to say, since he's done some experimentation with automating this with visual diffing, but I don't believe we got anything production ready there.

[AA-Turner]

@ericholscher Thanks!

By 'the docs' do you mean the Sphinx docs, or a custom docs test suite?

The Sphinx docs would be the easiest option - I could do that later today!

A

[ericholscher]

@AA-Turner I believe we were using these pages as a test: https://sphinx-rtd-theme.readthedocs.io/en/stable/demo/demo.html

There is a bunch of additional context in this thread: #9088 (comment) .

It also looks like @pradyunsg might have done some initial work to render a bunch of different themes here: sphinx-themes/sphinx-themes.org#70 -- so that could be a good start for visual diffing. It seems they include a "kitchen sink" option as well: https://sphinx-themes.org/sample-sites/sphinx-rtd-theme/kitchen-sink/paragraph-markup/. But I don't believe there is a live URL version of the sphinx 4.x branch, which could be interesting.

So I'd love to have an easy way to test the "top" themes, so perhaps:

• Sphinx docs themselves
• Alabaster
• RTD theme
• Furo
• Pydata

I think are the major ones. The "top 10" on sphinx-themes.org would also be another larger list if we wanted more.

[tk0miya]

if the html & css are exactly the same I imagine this would have the same rendering.

Unfortunately, this is not promised. The changes of docutils would cause changes in the output of Sphinx. Actually, docutils-0.17 changed the output of Sphinx. This is why we need to pin the docutils and test it manually before unpinning.

[pradyunsg]

Unfortunately, this is not promised.

Would it be useful to make this feasibile? IIUC, this would involve adding tests, that check the output of expanding all variables in the templates; which would effectively result in the validation of the HTML output in Sphinx's test suite.

Realistically, that's the bit that we want to test on Sphinx's end.

that could be a good start for visual diffing.

I'll pick this up, and tackle this over the weekend. :)

[tk0miya]

Would it be useful to make this feasibile? IIUC, this would involve adding tests, that check the output of expanding all variables in the templates; which would effective result in the validation of the HTML output into Sphinx' test suite.

Indeed, it would be helpful. Some discussion is needed to create it because we don't have the expected output for some structures.

[agjohnson]

@pradyunsg @AA-Turner last time we wanted to verify a docutils change, we manually built Sphinx output and then manually verified that nothing changed too drastically by eyeballing the two versions side by side. Out test matrix was larger last time, but we did test across several themes as well. I have at least one script to make this easier, we just put up the files at S3 for review.

I've also been down the visual diff testing path already. I do have a proof of concept ready here and could help get that to a place more testable with time. This project is using Happo and Cypress for visual diffing, but so far is only concerned with testing sphinx_rtd_theme. You can see a visual diff here:

readthedocs/test-sphinx-theme#6

I did hit the following issues with visual diff testing changes like this though:

• The last few docutils releases have mostly resulted in effective altered spacing of elements (at least for sphinx_rtd_theme). On a large page like the sphinx_rtd_theme demo docs, these spacing changes are cumulative and result in an unusable diff. A screenshot of the page can't be used for a visual diff because the cumulative changes output a diff where nearly 100% of the images mismatch.
• Dissecting the demo doc pages into sections and doing diff on individual sections is far more reliable. It's also fiddly to set up.
• The largest issue is the test matrix issue. sphinx_rtd_theme has an incredible large test matrix, as we still have to support multiple versions of python, sphinx, docutils, html 4/5, and test across browsers -- we aren't strict about it, but the product of this is 100+ test cases that we need to be aware of.

So, I'd suggest keep visual diff testing on the table for later, manual testing can be achieved far quicker.

[agjohnson]

I've updated one of the scripts we used in the past for our test matrix, tweaked to output for multiple themes instead. These builds all use the QA docs from sphinx_rtd_theme. Furo/pydata/sphinx_rtd_theme are all latest releases.

Output is available here:
https://readthedocs-static-dev.s3.us-east-2.amazonaws.com/qa/theme/index.html

Nothing surprising, but docutils 0.18 isn't well supported on themes just yet -- this is mostly limited to footnotes/citations.

[AA-Turner]

Thanks Anthony, this is really useful. I'll have a look over the next few days if there are any major areas of Sphinx that we can fix to cover the majority of the differences.

A

[agjohnson]

If I get a chance I'll put this code somewhere useful, but ping me if I can update this.

It seems each theme needs to be aware of the docutils 0.18 issues and should at least pin dependencies. I had started this discussion for sphinx_rtd_theme, but we don't have a release out yet.

[pradyunsg]

I'll pick this up, and tackle this over the weekend. :)

I haven't had time to pick this up, so... please don't feel obligated to move forward without me here. I'll update Furo's footnote styling as we get closer to Sphinx 5.0.

[AA-Turner]

Rebased to target 5.x.

A

[tk0miya]

Merged. Thank you!