Smaller/wrong search index with sphinx-build -j auto

[rogerbinns]

Describe the bug

Using the latest version and building with -j auto results in a considerably smaller searchindex,js and as a result most searches returning nothing. If I leave out -j then the searchindex is considerably larger and searches work as expected.

Some quick testing showed that Sphinx 6.0.0 did not have this problem while Sphinx 6.1.0 onwards does.

How to Reproduce

This fails:

sphinx-build -j auto  -n -b html -d build/doctrees -q -E . build/html

This works:

sphinx-build -n -b html -d build/doctrees -q -E . build/html

Environment Information

Platform:              linux; (Linux-5.19.0-29-generic-x86_64-with-glibc2.36)
Python version:        3.10.7 (main, Nov 24 2022, 19:45:47) [GCC 12.2.0])
Python implementation: CPython
Sphinx version:        6.1.3
Docutils version:      0.19
Jinja2 version:        3.0.3
Pygments version:      2.14.0

Sphinx extensions

No response

Additional context

Sphinx conf.py (almost empty!) and other files are at https://github.com/rogerbinns/apsw/tree/master/doc

[phlax]

im also hitting this and can confirm that downgrading to 6.0.1 resolves

[jbms]

I created a fix here: #11192