failing test run counts as successful

[finswimmer]

Hey,

I'm just starting to play around with mutation testing.

I'm under the impression, that mutmut thinks that a test suite run successfully and therefor a mutation survived, if the mutation that was introduced, leads to the situation, where the test suite doesn't run at all.

Here's my example where I saw this behavior

from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column


class Base(DeclarativeBase):
    pass


class User(Base):
    __tablename__ = "users"

    user_id: Mapped[int] = mapped_column(primary_key=True, autoincrement=False)
    name: Mapped[str]

Testfile:

from min_mut.model import User


def test_user():
    user = User(name="John Doe")

    assert user.name == "John Doe"

One of the mutation that mutmut introduces is:

@@ -8,6 +8,6 @@
 class User(Base):
     __tablename__ = "users"
 
-    user_id: Mapped[int] = mapped_column(primary_key=True, autoincrement=False)
+    user_id: Mapped[int] = mapped_column(primary_key=False, autoincrement=False)
     name: Mapped[str]

Applying this change to the code causes pytest to crash before it can invoke any test (which is expected).

====================================================================================================================== ERRORS ======================================================================================================================
_______________________________________________________________________________________________________ ERROR collecting tests/test_model.py _______________________________________________________________________________________________________
.venv/lib/python3.8/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
.venv/lib/python3.8/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.venv/lib/python3.8/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
.venv/lib/python3.8/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
.venv/lib/python3.8/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
.venv/lib/python3.8/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
.venv/lib/python3.8/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
.venv/lib/python3.8/site-packages/_pytest/pathlib.py:565: in import_path
    importlib.import_module(module_name)
/usr/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1014: in _gcd_import
    ???
<frozen importlib._bootstrap>:991: in _find_and_load
    ???
<frozen importlib._bootstrap>:975: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:671: in _load_unlocked
    ???
.venv/lib/python3.8/site-packages/_pytest/assertion/rewrite.py:178: in exec_module
    exec(co, module.__dict__)
tests/test_model.py:1: in <module>
    from min_mut.model import User
min_mut/model.py:8: in <module>
    class User(Base):
.venv/lib/python3.8/site-packages/sqlalchemy/orm/decl_api.py:835: in __init_subclass__
    _as_declarative(cls._sa_registry, cls, cls.__dict__)
.venv/lib/python3.8/site-packages/sqlalchemy/orm/decl_base.py:247: in _as_declarative
    return _MapperConfig.setup_mapping(registry, cls, dict_, None, {})
.venv/lib/python3.8/site-packages/sqlalchemy/orm/decl_base.py:328: in setup_mapping
    return _ClassScanMapperConfig(
.venv/lib/python3.8/site-packages/sqlalchemy/orm/decl_base.py:582: in __init__
    self._early_mapping(mapper_kw)
.venv/lib/python3.8/site-packages/sqlalchemy/orm/decl_base.py:369: in _early_mapping
    self.map(mapper_kw)
.venv/lib/python3.8/site-packages/sqlalchemy/orm/decl_base.py:1949: in map
    mapper_cls(self.cls, self.local_table, **self.mapper_args),
<string>:2: in __init__
    ???
.venv/lib/python3.8/site-packages/sqlalchemy/util/deprecations.py:281: in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
.venv/lib/python3.8/site-packages/sqlalchemy/orm/mapper.py:853: in __init__
    self._configure_pks()
.venv/lib/python3.8/site-packages/sqlalchemy/orm/mapper.py:1637: in _configure_pks
    raise sa_exc.ArgumentError(
E   sqlalchemy.exc.ArgumentError: Mapper Mapper[User(users)] could not assemble any primary key columns for mapped table 'users'
============================================================================================================= short test summary info ==============================================================================================================
ERROR tests/test_model.py - sqlalchemy.exc.ArgumentError: Mapper Mapper[User(users)] could not assemble any primary key columns for mapped table 'users'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
================================================================================================================= 1 error in 0.40s =================================================================================================================

Another situation I saw was, where a type hint like str | None was changed to str & None. The test suite doesn't run at all with this change, but mutmut insists the mutant survived.

fin swimmer

[finswimmer]

Here any return code not equal to 1 is interpreted as a successful test:

mutmut/mutmut/__init__.py

Line 874 in 53d4aa1

return returncode != 1

pytest returns a 2 in the above case. (https://docs.pytest.org/en/7.1.x/reference/exit-codes.html)

What's the reason for interpreting any return not equal to 1 as successful ? Shouldn't we just say, if it's 0, it was successful and all other cases the test failed?

[boxed]

5 is "no tests run" for pytest unfortunately. Which can happen for coverage driven runs.