ImportError does not result in collection error

[collielimabean]

This appears to have been introduced in 0.23.2 (0.21.x works):

1. pip3 install pytest pytest-asyncio
2. Define the following file (or test)

test_one.py

import asdf # nonexistent module

def test_abc():
  pass

3. pytest --import-mode=importlib will report no tests run under pytest-asyncio==0.23.2, but will show the ImportError under pytest-asyncio==0.21.1. Note that the default import mode (--import-mode=prepend) does not exhibit this problem.

Output when running under 0.23.2:

(.venv) williamjen@xps-32:~/Documents/import-error-pytest$ cat test_one.py 
import asdf

def test_abc():
    pass
(.venv) williamjen@xps-32:~/Documents/import-error-pytest$ pytest --import-mode=importlib
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/williamjen/Documents/import-error-pytest
plugins: asyncio-0.23.2
asyncio: mode=strict
collected 0 items                                                                                                                                                                                               

============================================================================================= no tests ran in 0.01s =============================================================================================
(.venv) williamjen@xps-32:~/Documents/import-error-pytest$

Output when running under 0.21.1:

(.venv) williamjen@xps-32:~/Documents/import-error-pytest$ pytest --import-mode=importlib
============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.10.12, pytest-7.4.3, pluggy-1.3.0
rootdir: /home/williamjen/Documents/import-error-pytest
plugins: asyncio-0.21.1
asyncio: mode=strict
collected 0 items / 1 error                                                                                                                                                                                     

==================================================================================================== ERRORS =====================================================================================================
_________________________________________________________________________________________ ERROR collecting test_one.py __________________________________________________________________________________________
ImportError while importing test module '/home/williamjen/Documents/import-error-pytest/test_one.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
test_one.py:1: in <module>
    import asdf
E   ModuleNotFoundError: No module named 'asdf'
============================================================================================ short test summary info ============================================================================================
ERROR test_one.py
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================

My suspicion is the CollectError caught in a214c3e since the ImportError appears to be propagated via a CollectError, but I'm not sure.

[seifertm]

Thanks for the investigation. This seems to be connected to #713 .

The pytest-asyncio currently creates "dynamic" fixtures by adding them to the Python object, for example a module. Once the module is collected by pytest, the fixture is evaluated as usual. This approach is internally used by pytest as well and leaves the complicated fixture evaluation to pytest rather than having to reproduce in pytest-asyncio.

Unfortunately, pytest-asyncio currently adds those fixtures as part of the pytest_collectstart hook. When trying to add a fixture to a module, that module is imported and may raise errors like pytest._outcomes.Skip or pytest.CollectError. The pytest_collectstart hook isn't supposed to handle those kinds of exceptions, so pytest-asyncio tries to do it. That effort seems to be unsuccessful so far.

Given that fixing one bug leads to another bug, I think pytest-asyncio should look for an entirely different place to add the fixtures rather than in pytest_collectstart. There are chances that, in the future, pytest will provide a way to add fixtures to nodes, but until then pytest-asyncio needs to find a better workaround.

[seifertm]

@collielimabean This should be resolve with pytest-asyncio v0.23.3