test_logging_in_ParallelTasks fails with Python 3.8 (macos x 10.9.5)

[jfbu]

====================================================== test session starts ======================================================
platform darwin -- Python 3.8.0, pytest-5.3.0, py-1.8.0, pluggy-0.13.1 -- /Users/path/to/venv38/bin/python3
cachedir: .pytest_cache
libraries: Sphinx-3.0.0+/8c7faed6f, docutils-0.15.2

[...]
_________________________________________________ test_logging_in_ParallelTasks _________________________________________________

app = <SphinxTestApp buildername='html'>, status = <_io.StringIO object at 0x1081784c0>
warning = <_io.StringIO object at 0x108178550>

    @pytest.mark.xfail(os.name != 'posix', reason="Not working on windows")
    def test_logging_in_ParallelTasks(app, status, warning):
        logging.setup(app, status, warning)
        logger = logging.getLogger(__name__)
    
        def child_process():
            logger.info('message1')
            logger.warning('message2', location='index')
    
        tasks = ParallelTasks(1)
>       tasks.add_task(child_process)

tests/test_util_logging.py:288: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
sphinx/util/parallel.py:96: in add_task
    self._join_one()
sphinx/util/parallel.py:120: in _join_one
    self._procs[newtid].start()
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py:121: in start
    self._popen = self._Popen(self)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:224: in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py:283: in _Popen
    return Popen(process_obj)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:32: in __init__
    super().__init__(process_obj)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_fork.py:19: in __init__
    self._launch(process_obj)
/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/popen_spawn_posix.py:47: in _launch
    reduction.dump(process_obj, fp)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

obj = <Process name='Process-1' parent=7430 initial>, file = <_io.BytesIO object at 0x10818adb0>, protocol = None

    def dump(obj, file, protocol=None):
        '''Replacement for pickle.dump() using ForkingPickler.'''
>       ForkingPickler(file, protocol).dump(obj)
E       AttributeError: Can't pickle local object 'ParallelTasks.add_task.<locals>.<lambda>'

/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py:60: AttributeError
--------------------------------------------------- Captured stdout teardown ----------------------------------------------------
# testroot: root
# builder: html
# srcdir: /private/var/folders/np/4_9pppfs54xdv9m79m7rwx4w0000gn/T/pytest-of-johndoe/pytest-1/root
# outdir: /private/var/folders/np/4_9pppfs54xdv9m79m7rwx4w0000gn/T/pytest-of-johndoe/pytest-1/root/_build/html
# status: 
Running Sphinx v3.0.0+/8c7faed6f
loading pickled environment... done

# warning: 
WARNING: while setting up extension sphinx.domains.changeset: directive 'deprecated' is already registered, it will be overridden
WARNING: while setting up extension sphinx.domains.changeset: directive 'versionadded' is already registered, it will be overridden
WARNING: while setting up extension sphinx.domains.changeset: directive 'versionchanged' is already registered, it will be overridden
WARNING: while setting up extension sphinx.domains.math: role 'eq' is already registered, it will be overridden

======================================================= warnings summary ========================================================
/Users/path/to/venv38/lib/python3.8/site-packages/html5lib/_trie/_base.py:3
  /Users/path/to/venv38/lib/python3.8/site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
    from collections import Mapping

tests/test_autodoc.py::test_autodoc_imported_members
tests/test_autodoc.py::test_autodoc_imported_members
tests/test_autodoc.py::test_autodoc_imported_members
tests/test_autodoc.py::test_autodoc_imported_members
tests/test_autodoc.py::test_autodoc_imported_members
tests/test_autodoc.py::test_autodoc_imported_members
  /Users/path/to/venv38/bin/../lib/python3.8/site.py:429: DeprecationWarning: 'U' mode is deprecated
    fp = open(filename, "rU")

tests/test_highlighting.py::test_add_lexer
  /Users/path/to/sphinx/sphinx/application.py:987: RemovedInSphinx40Warning: app.add_lexer() API changed; Please give lexer class instead instance
    warnings.warn('app.add_lexer() API changed; '

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=============================== 1 failed, 1300 passed, 1 skipped, 8 warnings in 144.09s (0:02:24) ===============================
make: *** [Makefile:72: test] Error 1

Environment info

• OS: macosx 10.9.5
• Python version: 3.8.0 (from python.org installer)
• Sphinx version: master at 8c7faed

Additional context

The python.org says it supports mac os x starting at 10.9

I proceeded from an "editable" install of Sphinx in a virtual environment.

edit: I forgot to say that testing Sphinx under Python 3.7.5 installed similarly succeeds. And the Python 3.8 announcement mentions Parallel filesystem cache for compiled bytecode files
which may be or not relevant.

[tk0miya]

The default setting of multiprocessing on macOS has been changed since python 3.8. Now it uses spawn model instead of fork. As a result, we can't use lambda and internal function to invoke multiprocessing. If my understanding correct, there are no way to avoid that safely.
https://bugs.python.org/issue33725

Note: we can switch the strategy of multiprocessing to "fork" model again via multiprocessing.set_start_method('fork'). But it is not unsafe operation...
https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method

[tk0miya]

This is a smallest example to cause pickling error on multiprocessing. As reported, it works fine before python 3.8.

import multiprocessing

def main():
    def proc():
        pass

    p = multiprocessing.Process(target=proc)
    p.start()
    p.join()

# If we uncommented this, it works fine. But unsafe.
#multiprocessing.set_start_method('fork')
main()

[tk0miya]

refs: #6803

[jfbu]

Thanks for detailed explanations and minimal example. For some reasons I am sticking with an old macos x (10.9.5) so I wasn't sure whether the issue could be related to old system. But it appears it applies to latest macos x too. And I had not seen #6803, now I mark this as duplicate.

[tk0miya]

This is not related with version of macOS. Python 3.8 switches the start method to "spawn" on all macOS systems. see https://github.com/python/cpython/pull/13603/files.

[tk0miya]

Fixed by #6879 !