Avoid zombie processes on parallel build fail

[felixvd]

Subject: Terminate processes correctly when parallel build fails

Feature or Bugfix

• Bugfix

Purpose

On our build servers, we have had recurring (every 1-4 weeks) and hard-to-reproduce issues with sphinx builds that do not terminate. We have applied the change in this MR and have seen no reoccurrence of the error.

Detail

Without this change, this error message used to appear in our build logs at rare times:

Exception occurred:
  File "/usr/lib/python3.9/multiprocessing/popen_fork.py", line 66, in _launch
    self.pid = os.fork()
OSError: [Errno 12] Cannot allocate memory

I assume that when one of the threads dies, it cannot be joined and the build hangs.

Sadly, I lost most of the logs for and description of the original issue. All I know is that this change fixed the issue for us to the best of our knowledge.

I apologize for the lack of documentation, but I hope this can be merged nonetheless.

Relates

• A similar problem was reported and fixed in Use finally to terminate parallel processes #10952, but it did not cover this instance of _join_one.

cc @cielavenir

[picnixz]

I'm wondering whether we should re-raise or not.

[AA-Turner]

Thanks Felix!

A

[felixvd]

Thanks for the quick merge and your maintaining this package Adam. It’s much appreciated.

…

On Sun, Feb 4, 2024 at 8:57 Adam Turner ***@***.***> wrote: Thanks Felix! A — Reply to this email directly, view it on GitHub <#11923 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCTLOL2V4IZTCEUUHSZ2VDYR3FE3AVCNFSM6AAAAABCSTZRO6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRVGQ4TANJYGQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>