RTD is pinning "sphinx < 2" somehow

[spookylukey]

This relates to the recent problems with docutils >= 0.18 e.g. #8663

It looks like one reason why many builds are failing is that RTD is pinning Sphinx to < 2. This was happening in my old builds, where I hadn't specified any Sphinx version, and I've noticed that even after I've upgraded Sphinx, using docs/requirements.txt in .readthedocs.yaml as instructed, my builds are still doing pip install sphinx<2. This is followed by pip install docs/requirements.txt, so the builds now succeed, but the first installation is now useless and inefficient.

In my builds, I had never pinned to sphinx < 2 , but this was chosen for me for some reason, perhaps because it was an old project, as suggested in this comment. I had assumed this would stop happening after upgrading my .readthedocs.yaml file, but it hasn't.

Example:

https://readthedocs.org/projects/parsy/builds/15271965/ - this is from after I fixed the build, so it completes successfully, but still includes the sphinx<2 pinning in the line:

/home/docs/checkouts/readthedocs.org/user_builds/parsy/envs/latest/bin/python -m pip install --upgrade --no-cache-dir mock==1.0.1 pillow==5.4.1 alabaster>=0.7,<0.8,!=0.7.5 commonmark==0.8.1 recommonmark==0.5.0 sphinx<2 sphinx-rtd-theme<0.5 readthedocs-sphinx-ext<2.2

Later on it does:

/home/docs/checkouts/readthedocs.org/user_builds/parsy/envs/latest/bin/python -m pip install --exists-action=w --no-cache-dir -r docs/requirements.txt

This project is not a Python 2 project, and I've configured it to build with Python 3 only, so I'm at a loss now to explain what is triggering this behaviour. It seems like at the very least RTD could be more efficient here.

My .readthedocs.yaml contains:

version: 2

build:
  image: latest

python:
  version: 3.8
  install:
  - requirements: docs/requirements.txt
  - method: setuptools
    path: .

and docs/requirements.txt contains:

sphinx==4.3.0

More examples:

• A different project of mine: https://readthedocs.org/projects/django-functest/builds/15271985/
• Someone else's project (so it's not just me doing something wrong!): https://readthedocs.org/projects/obiba-opal/builds/15232551/

[astrojuanlu]

Hi @spookylukey - that's correct, we pin Sphinx < 2 for projects created before October 2020 as explained in https://docs.readthedocs.io/en/stable/builds.html?highlight=sphinx#python, even though, as you discovered, you can override that with the proper configuration.

If this is a concern for you, we can enable the USE_SPHINX_LATEST flag for your project, like we did in #8324.

readthedocs.org/readthedocs/doc_builder/python_environments.py

Lines 506 to 511 in 3c4b495

	requirements.extend([
	self.project.get_feature_value(
	Feature.USE_SPHINX_LATEST,
	positive='sphinx',
	negative='sphinx<2',
	),

[spookylukey]

Hi @astrojuanlu - thanks for the reply. I can see why you took the decision to do this pinning, but this piece of information is buried quite deep in the docs. Even though I was looking for it, I didn't find it. I had been scouring the configuration docs to find it, and it wasn't mentioned in this blog post - https://blog.readthedocs.com/build-errors-docutils-0-18/

It is also very non-intuitive - I also have some settings on the web interface which control how my project is built, I then have a configuration file which overrides that, and then I have another invisible bit of configuration, which depends on when my project was first created, which is very surprising, and I can think of no other software that works like that. This behaviour means that I can copy the configuration from one project to another, and RTD doesn't behave the same, which is quite baffling.

So perhaps:

• there should be a note in the configuration file docs - https://docs.readthedocs.io/en/stable/config-file/v2.html#sphinx
• there could be a note in the blog post about it https://blog.readthedocs.com/build-errors-docutils-0-18/
• there could be a note in the web interface, perhaps under "Advanced settings", indicating that this mode is enabled, even if I'm not allowed to change it directly. Even better would be an opt out button (even if it is one way, that would be fine, because we can still configure what we need by the config file).

In the context of the recent docutils breakage, this behaviour is particularly confusing - I'm not specifying a Sphinx version at all, which I expect to be interpreted as "I don't care, just give me the default", but in this case RTD gives me a default that doesn't work.

---

Regarding my own builds - yes, please do change the setting to enable most recent Sphinx, for these projects:

django-ftl
django-functest
django-paypal
elm-fluent
fluent-compiler
parsy
py-moneyed
signupto

Thanks!

[astrojuanlu]

Thanks for the writeup @spookylukey , these are all good suggestions. I'm leaving the issue open so I don't forget about them.

I already enabled the flag for all your projects, please trigger a build for them just to check that everything works.

[astrojuanlu]

there could be a note in the web interface, perhaps under "Advanced settings", indicating that this mode is enabled, even if I'm not allowed to change it directly. Even better would be an opt out button (even if it is one way, that would be fine, because we can still configure what we need by the config file).

This suggestion is also good, but our policy at the moment is to avoid touching the current UI, since we are working on a complete redesign.

For the other two, I have opened two pull requests on the corresponding repositories.

[astrojuanlu]

Both PRs were merged, closing.