Sphinx silently drops warnings when libraries like 'airflow' or 'mlflow' are imported in the middle of the build

[gibsondan]

Describe the bug

I'm finding that sphinx does not emit warnings that it should be emitting.

For example, there is a typo here: https://github.com/dagster-io/dagster/blob/master/docs/sphinx/sections/api/apidocs/execution.rst#L56

ReeconstructablePipeline should be ReconstructablePipeline (there is an extra 'e' character), which should produce a Sphinx warning since the class does not exist.

Confusingly, I'm finding that the expected warnings start appearing when I reduce the number of libraries in my environment, even when the libraries that I remove are not specifically required by the sphinx build. I noticed this when I tried to start using autodoc_mock_imports to reduce our imports - suddenly, a bunch of unrelated warnings started appearing.

It does not seem that including a specific library causes the problem (I verified this by attempting to bisect the problem by reducing libraries one by one) - once I reduce the number of libraries in my virtual environment enough, the expected warnings begin appearing again during the build.

See specific reproduction instructions below. Thanks for any help you can provide.

How to Reproduce

(in a fresh virtualenv with pip 19.2.3 installed)

git clone https://github.com/dagster-io/dagster.git --branch sphinxbugreport --depth 1
cd dagster/docs
pip install -r docs-requirements.txt
cd sphinx; make json SPHINXOPTS="-W";

(sphinx will build successfully with no warnings, despite having the ReeconstructablePipeline typo mentioned above that should generate a warning and thus an error due to the -W flag)

Now, to make the warning appear, follow these steps to remove a bunch of libraries that are not actually needed by the sphinx build (it does not seem that there's one individual library that specifically causes the problem - I attempted to find one by bisecting, but it seems that it's just a matter of removing enough libraries):

$ cd ..
$ cat toremove | xargs pip uninstall -y
$ cd sphinx; make json SPHINXOPTS="-W";

The sphinx build will now correctly create a warning for the typo:

Warning, treated as error:
autodoc: failed to import class 'ReeconstructablePipeline' from module 'dagster.core.definitions.reconstructable'; the following exception was raised:
Traceback (most recent call last):
  File "/Users/dgibson/.pyenv/versions/3.8.3/envs/sphinxreport/lib/python3.8/site-packages/sphinx/util/inspect.py", line 393, in safe_getattr
    return getattr(obj, name, *defargs)
AttributeError: module 'dagster.core.definitions.reconstructable' has no attribute 'ReeconstructablePipeline'

Expected behavior

Sphinx will always produce warnings for missing classes

Your project

https://github.com/dagster-io/dagster/tree/sphinxbugreport/docs/sphinx

Screenshots

No response

OS

Mac

Python version

3.8.3

Sphinx version

3.5.2

Sphinx extensions

sphinx.ext.autodoc, sphinx.ext.ifconfig, sphinx.ext.intersphinx, sphinx.ext.viewcode, nbsphinx, sphinx.ext.autosectionlabel

Extra tools

No response

Additional context

No response

[gibsondan]

Just tried upgrading to Sphinx 4.2.0 and confirmed that warnings are still missing after the upgrade.

[gibsondan]

Another useful bit of information - the warnings still show up if you run sphinx with '-vv' and filter for WARNING (they just aren't output by sphinx normally, and they don't stop the build if you run with '-W'):

e.g. in the bug report above, without making any changes,

make json SPHINXOPTS="-W"

succeeds without logging any warnings (or failing), but:

make json SPHINXOPTS="-W -vv" 2>&1 | grep WARNING

shows the warnings:

/Users/dgibson/dagster/docs/sphinx/sections/api/apidocs/experimental.rst:72: (WARNING/2) Title underline too short.
/Users/dgibson/dagster/docs/sphinx/sections/api/apidocs/internals.rst:7: (WARNING/2) Line block ends without a blank line. [filtered system message]
/Users/dgibson/dagster/docs/sphinx/sections/api/apidocs/internals.rst:11: (WARNING/2) Line block ends without a blank line. [filtered system message]
/Users/dgibson/dagster/python_modules/dagster/dagster/core/instance/init.py:docstring of dagster.core.instance.DagsterInstance:13: (WARNING/2) Include file ‘/Users/dgibson/dagster/docs/docs/sections/deploying/postgres_dagster.yaml’ not found or reading it failed [filtered system message]
/Users/dgibson/dagster/docs/sphinx/sections/api/apidocs/internals.rst:173: (WARNING/2) Title underline too short.
/Users/dgibson/dagster/docs/sphinx/sections/api/apidocs/io-managers.rst:53: (WARNING/2) Title underline too short.

Similarly, the import error from autodoc seems to not be raised even though it can be found in the verbose logs.