Make the distutils/setuptools integration an opt-in for projects

[pradyunsg]

Is your feature request related to a problem? Please describe.

Sphinx currently injects a "magical" setup.py build_sphinx command into setuptools-based projects. While this is likely be useful for some user workflows, recently this has been a major source of confusion in the Python ecosystem, due to some folks who are repackaging a substantial number of open source Python packages.

These repackagers have assumed that all Python projects that use setuptools and Sphinx also use this integration -- which is not true. Most users are not aware of this integration and do not use it or support it. By making projects explicitly opt-in to this integration, it would help avoid such confusion and problems due to repackagers trying to build documentation using this integration and that not working -- since it's not a way that the specific project supports for building documentation (eg: if they use any non-setuptools build backend).

Further, this also puts setuptools in a uniquely priviledged position with Sphinx -- something that Python packaging tooling has been actively working to avoid and move away from (see pyproject.toml based builds, introduced in PEP 517 / PEP 518).

Describe the solution you'd like

Move the setuptools integration into a separate package on PyPI. We can also add it as an optional dependency of Sphinx (via the extras mechanism), so that users who want this can use sphinx[setuptools] in their requirements.txt or pip install invocations.

Describe alternatives you've considered

Not doing anything, which... works. 🤷🏽‍♂️

I think this is suboptimal though -- see https://github.com/search?q=build_sphinx+is%3Aissue&type=issues -- there are many issues filed on the premise that this integration means that projects should actively support this, which is a generally harmful suggestion to make IMO. These assumptions break down for non-setuptools-based projects as well.

Additional context

I guess, it's worth noting that a certain individual has been filed 100s of issues against various Python projects regarding this exact issue -- demanding that the entire ecosystem support this integration as a way to build their documentation (see the search above).

[astrojuanlu]

Recent conversation on the topic (that got derailed, anyway) on one of such issues #9517 (comment) . I should have opened a new issue about it, thanks @pradyunsg. +1 from me.

[hukkin]

+1 to this.

An alternative low effort solution would be to simply deprecate and eventually remove this feature entirely in favor of a direct sphinx-build invocation (possibly via tox). My understanding is that setuptools is actively trying to remove all python setup.py <cmd> invocations, so IMO would make sense to do the same thing here.

[astrojuanlu]

I was about to open a new issue, but I'll continue the conversation here instead.

Paul Ganssle has written it better than I could: https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

TL;DR: The setuptools team no longer wants to be in the business of providing a command line interface and is actively working to become just a library for building packages.
This does not mean that setuptools itself is deprecated, or that using setup.py to configure your package builds is going to be removed. The only thing you must stop doing is directly executing the setup.py file — instead delegate that to purpose-built or standards-based tools, preferably those that work with any build backend.
...
You may still find advice all over the place that involves invoking setup.py directly, but unfortunately this is no longer good advice because as of the last few years all direct invocations of setup.py are effectively deprecated in favor of invocations via purpose-built and/or standards-based CLI tools like pip, build and tox.

Related: #9517, #9514, #9378, #9279, #3837.

[tk0miya]

Now Sphinx uses setuptools to search 3rd party builders and themes via pkg_resources. It has not been used only for build_sphinx.

About build_sphinx, I don't have a strong opinion on removal. So +0 for your proposal.

[pradyunsg]

Sphinx uses setuptools to search 3rd party builders and themes via pkg_resources

This is for entry points, right?

Those can be replaced with importlib.metadata and its backport, which are also much faster, use less memory and (>= 3.8) is a part of the standard library.

[tk0miya]

Sounds good. As a first step, we should replace pkg_resources by importlib.metadata and its backport lib. Then we can consider how to deprecate dependency of setuptools.

[takluyver]

I've opened #10007 to get rid of pkg_resources. I think there's nothing else in Sphinx that actually relies on setuptools - the setup.py build_sphinx command is implemented directly on distutils (another reason to get rid of it :-).

[pradyunsg]

Here's the list of reasons I can think of, to drop the build_sphinx command:

• it extends the setup.py as a script interface, which is considered deprecated by the maintainers of setuptools and distutils.
• it is built using distutils, which has been deprecated starting Python 3.10, according to PEP 632.
• it does not have feature-parity with the sphinx-build command line script, which caused subtle issues and additional maintenance workload (both, for Sphinx maintainers as well as downstream users) due to these behaviour differences.
• it puts distutils/setuptools in a priviledged position. Sphinx does not provide integration for all the other build-backends that Python packaging tooling can now support. It is better have a nuetral position and avoid influencing the choice of build-backends in that manner. The alternative is to integrate with other build-backends, which is both non-traceable, and something that can be done externally.

[pradyunsg]

Then we can consider how to deprecate dependency of setuptools.

There's broadly two options here, as I see it:

1. Keep the code, but drop the automatic registration of the build_sphinx endpoint. This means only the projects that follow the example on https://www.sphinx-doc.org/en/master/usage/advanced/setuptools.html#using-setuptools-integration (which explicitly registers the command) will continue working.

   • good: This should be the least disruptive approach here.
   • good: Only the projects that explicitly register the command in their setup.py (i.e. opt-in) will get the integration.
   • bad: This does not remove the maintainance burden this integration might serve for Sphinx's maintainers.
   • bad: See previous comment.

2. Drop the code and push users to use sphinx-build directly.

   • See previous comment for good reasons to do this.

My earlier suggestion of moving the code from Sphinx to a separate package was... not a good one. Other than being incompatible with projects that followed the documented guidance, it also doesn't achieve the main goal: it still does not make it opt-in at the project level -- the hook can still be registered without the project opting in, by installing this package in the build environment. The whole premise of this issue is that it shouldn't be possible for a redistributor to build documentation through this integration for projects that have not opted in explicitly for this integration.

Option 1 makes the opt-in happen explicitly, as documented. Option 2 removes the possibility of opting in, by removing the functionality entirely.

[pradyunsg]

I've opened #10008, in case we want to go with option 2. :)

[takluyver]

If this was one of my projects, I'd be inclined to do a deprecation cycle, issue warnings etc. It looks like Sphinx is developed more semver style, where breaking things in major versions is OK. Given that, I'd be inclined to remove it in the next major version.

[tk0miya]

Our deprecation policy is here: https://github.com/sphinx-doc/sphinx/blob/4.x/doc/internals/release-process.rst#deprecation-policy
So we can't accept the removal suddenly. Let's start emitting a warning since Sphinx-5.0 and remove it on Sphinx-7.0.

[pradyunsg]

Sounds fair, if we do want to go with option 2 (drop the functionality)!

If we went with option 1 (stop the automated registration of the build_sphinx command), would it be OK to not have a multi release deprecation cycle, since it is not possible to differentiate between use via the automated registration vs direct use as documented?

[tk0miya]

As a first step of the deprecation, we should keep the compatibility (without warnings). If my understanding is correct, option 1 affects users, right?

[takluyver]

Broadly speaking, yes: option 1 would mean setup.py build_sphinx only worked if a project had put something in setup.py to make it work. At present, it can work even without anything in setup.py to explicitly enable it.

[tk0miya]

I'm not sure the advantage of option 1 compared with the current implementation. IMO, it does not help the removal of the build_sphinx command in the future. So adding a warning to the command is a better way, I think.

[AA-Turner]

The setuptools integration is deprecated and will be removed in Sphinx 7.

A