Set multiprocessing start method to fork

[samdoran]

Subject: Set multiprocessing start method to fork

Feature or Bugfix

• Bugfix

Purpose

Explicitly set the multiprocessing start method to fork since the code depends on this behavior.

Detail

Since Python 3.8, the default start method was changed to spawn on macOS, which is incompatible with how Sphinx handles parallelization. The change was made in macOS not due to a new security concern but because Apple decided to prevent potential problems with calling something between fork() and exec() by failing early. This problem is not unique to macOS or even Python, and depends on the implementation.

From looking at the use of multiprocessing in Sphinx, I don't believe this will cause issues on macOS.

Relates

• Fix #6803: Disable parallel build on macOS and py38+ #6879

[tk0miya]

Thank you for the proposal. It would be exciting if we can support a parallel build on macOS again!

From looking at the use of multiprocessing in Sphinx, I don't believe this will cause issues on macOS.

Before merging this, could you let us know why do you think so? I'm not expert of the internal Python and macOS. So I can't determine why Sphinx will not cause issues.

[samdoran]

Before merging this, could you let us know why do you think so? I'm not expert of the internal Python and macOS. So I can't determine why Sphinx will not cause issues.

The problem is that calling into the Objective-C runtime between fork() and exec() is unsafe. This has always been the case. Apple chose to make calls like this fail early and deterministically rather than (potentially) failing later and in unpredictable ways.

From looking at the system calls using dtruss that Sphinx makes as well as testing parallel builds on macOS, Sphinx does not run into this, explicitly or implicitly.

[tk0miya]

Thank you for your detailed explanation. I read https://bugs.python.org/issue33725 again and understand the problem (maybe). Indeed, Sphinx does not do anything between fork and exec. Okay, let's see what happens with this.

[samdoran]

Thank you!