doc build fails on Python 3.8 + MacOS

[weakcamel]

SUMMARY

Documentation doesn't currently build on MacOS with Python3.8.

This is due to sphinx-doc/sphinx#6803 which is not fixed in the sphinx version used by ansible documentation build system (it's frozen to 2.1.2)

ISSUE TYPE

• Documentation Report

COMPONENT NAME

docs/docsite/requirements.txt

ANSIBLE VERSION

ansible 2.9.11
  config file = None
  configured module search path = ['/Users/waldekm/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.9.11/libexec/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.5 (default, Jul 21 2020, 10:48:26) [Clang 11.0.3 (clang-1103.0.32.62)]

CONFIGURATION

# empty output

OS / ENVIRONMENT

$ uname -rs
Darwin 19.6.0

$ python --version
Python 3.8.5

ADDITIONAL INFORMATION

Error log

$ make webdocs
(cd docs/docsite/; CPUS=8 /Applications/Xcode.app/Contents/Developer/usr/bin/make docs)
../../hacking/build-ansible.py collection-meta --template-file=../templates/collections_galaxy_meta.rst.j2 --output-dir=rst/dev_guide/ ../../lib/ansible/galaxy/data/collections_galaxy_meta.yml
../../hacking/build-ansible.py document-config --template-file=../templates/config.rst.j2 --output-dir=rst/reference_appendices/ ../../lib/ansible/config/base.yml
mkdir -p rst/cli
../../hacking/build-ansible.py generate-man --template-file=../templates/cli_rst.j2 --output-dir=rst/cli/ --output-format rst ../../lib/ansible/cli/*.py
[WARNING]: You are running the development version of Ansible. You should only run Ansible from "devel" if you are modifying the Ansible engine, or
trying out features under development. This is a rapidly changing source of code and can become unstable at any point.
../../hacking/build-ansible.py document-keywords --template-dir=../templates --output-dir=rst/reference_appendices/ ./keyword_desc.yml
if expr "2.11.0.dev0" : '.*[.]dev[0-9]\{1,\}$' &> /dev/null; then \
		../../hacking/build-ansible.py docs-build base -o rst ;\
	else \
		../../hacking/build-ansible.py docs-build full -o rst ;\
	fi
ERROR:antsibull:func=write_rst:mod=antsibull.write_docs:nonfatal_errors=['1 validation error for PluginDocSchema\ndoc -> options -> cache -> deprecated -> removed_from_collection\n  field required (type=value_error.missing)']:plugin_name=ansible.builtin.script:plugin_type=inventory|ansible.builtin.script did not return correct DOCUMENTATION.  An error page will be generated.
ERROR:antsibull:func=write_rst:mod=antsibull.write_docs:nonfatal_errors=['18 validation errors for PluginDocSchema\ndoc -> options -> ca_path -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> ca_path -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> follow_redirects -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> follow_redirects -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> force -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> force -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> force_basic_auth -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> force_basic_auth -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> http_agent -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> http_agent -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> timeout -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> timeout -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> unix_socket -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> unix_socket -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> unredirected_headers -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> unredirected_headers -> ini -> 1 -> section\n  field required (type=value_error.missing)\ndoc -> options -> use_gssapi -> ini -> 0 -> key\n  field required (type=value_error.missing)\ndoc -> options -> use_gssapi -> ini -> 1 -> section\n  field required (type=value_error.missing)']:plugin_name=ansible.builtin.url:plugin_type=lookup|ansible.builtin.url did not return correct DOCUMENTATION.  An error page will be generated.
../bin/testing_formatter.sh
../bin/testing_formatter.sh: line 38: ../docsite/rst/dev_guide/testing/sanity/index.rst: No such file or directory
CPUS=8 /Applications/Xcode.app/Contents/Developer/usr/bin/make -f Makefile.sphinx html
sphinx-build -M html "rst" "_build" -j 8 -n -w rst_warnings
Running Sphinx v2.1.2
making output directory... done
loading intersphinx inventory from https://docs.python.org/2/objects.inv...
loading intersphinx inventory from https://docs.python.org/3/objects.inv...
loading intersphinx inventory from http://jinja.palletsprojects.com/objects.inv...
intersphinx inventory has moved: http://jinja.palletsprojects.com/objects.inv -> https://jinja.palletsprojects.com/en/2.11.x/objects.inv
loading intersphinx inventory from https://docs.ansible.com/ansible/2.10/objects.inv...
loading intersphinx inventory from https://docs.ansible.com/ansible/2.9/objects.inv...
loading intersphinx inventory from https://docs.ansible.com/ansible/2.8/objects.inv...
loading intersphinx inventory from https://docs.ansible.com/ansible/2.7/objects.inv...
loading intersphinx inventory from https://docs.ansible.com/ansible/2.6/objects.inv...
loading intersphinx inventory from https://docs.ansible.com/ansible/2.5/objects.inv...
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 463 source files that are out of date
updating environment: 463 added, 0 changed, 0 removed
reading sources... [  5%] 404 .. collections/ansible/builtin/config_lookup
Exception occurred:
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'Builder._read_parallel.<locals>.merge'
The full traceback has been saved in /var/folders/f1/svj655pj0mj11j240_yh47840000gn/T/sphinx-err-w96ydjs6.log, if you want to report the issue to the developers.
Please also report this if it was a user error, so that a better error message can be provided next time.
A bug report can be filed in the tracker at <https://github.com/sphinx-doc/sphinx/issues>. Thanks!
make[2]: *** [html] Error 2
make[1]: *** [htmldocs] Error 2
make: *** [webdocs] Error 2

[weakcamel]

For the record, using the latest sphinx (3.2.1) completed successfully, although with plenty of warnings about the content, e.g.

reading sources... [100%] user_guide/windows_winrm
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:163: WARNING: duplicate term description of any_errors_fatal, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:166: WARNING: duplicate term description of become, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:169: WARNING: duplicate term description of become_exe, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:172: WARNING: duplicate term description of become_flags, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:175: WARNING: duplicate term description of become_method, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:178: WARNING: duplicate term description of become_user, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:181: WARNING: duplicate term description of check_mode, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:184: WARNING: duplicate term description of collections, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:193: WARNING: duplicate term description of connection, other instance in reference_appendices/playbooks_keywords
/Users/weakcamel/git/ansible-weakcamel/docs/docsite/rst/reference_appendices/playbooks_keywords.rst:196: WARNING: duplicate term description of debugger, other instance in reference_appendices/playbooks_keywords
...

[samdoran]

cc @abadger

[abadger]

@acozine @samccann would it be okay to update the sphinx version? I don't know why it's pinned right now.

[samccann]

@abadger it's pinned because of a note that says rstcheck only works up to 1.7.9. So we'd have to:

• bump it up to 2.2.2 (the version I think that has this fix)

• test locally, on a mac, and verify it doesn't throw any new errors (sphinx updates in the past have required some other fixes to go with it)

• test rstcheck to determine that note is no longer valid with modern version of rstcheck etc.

• update the jenkins build to pin to 2.2.2 and verify everything works

• decide 'how far back' this goes - do we just fix 2.11? fix all supported docs (back to 2.8)? etc.