inconsistent behavior report vs collect in Pytest 7 (corner case)

[stanislavlevin]

test example should include two test functions having the same name (this is just an example):

def test_1():
    assert False

def test_2():
    assert True

def test_1():
    assert True

Pytest 7:

platform linux -- Python 3.10.2, pytest-7.0.1, pluggy-1.0.0 -- /somedir/python
cachedir: .pytest_cache
rootdir: somedir, configfile: setup.cfg
collected 2 items                                                                    

test.py::test_1 PASSED                                                         [ 50%]
test.py::test_2 PASSED                                                         [100%]

Pytest 6:

platform linux -- Python 3.10.2, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /somedir/python
cachedir: .pytest_cache
rootdir: somedir, configfile: setup.cfg
collected 2 items                                                                    

test.py::test_2 PASSED                                                         [ 50%]
test.py::test_1 PASSED                                                         [100%]

With Pytest 7 tests are ordered by definition ignoring possible re-definition.
I can't find documented behavior for duplicated test functions, so feel free to close this ticket if this is something not supported.

[nicoddemus]

Hi @stanislavlevin,

Seems related to #9144, and is considered a feature.

Note that of course pytest only "sees" the second test_1 definition, just the order of the items changed.

Closing this then. 👍

[stanislavlevin]

Hi, @nicoddemus, thank you for reply.

I expected that the order of report == the order of collect. Was I wrong?

[nicoddemus]

Not sure, I'm not familiar with those changes in particular.

cc @asottile @bluetech

[asottile]

in pytest 6 there was an error prone attempt to find the location of the test by loading the tests' source and traversing through the ast and source

in pytest 7 the ordering is what cpython tells us it is (due to dictionaries maintaining insertion order and the module ~essentially being an object with __dict__ having its members)

the change in behaviour "makes sense" here -- but the code seems wrong (duplicate definitions are going to clobber themselves) -- so to me this seems "garbage in garbage out"

[RonnyPfannschmidt]

@asottile i believe the confusion comes from the fact that a later defined "replacement" will retain the order spot from the earlier insert

im a bit on the edge about deciding whether its a regression, as it practically only hits when that particular type of user error happens

a warning about that case may help people but ultimately its a thing a linter helps with

[nicoddemus]

but ultimately its a thing a linter helps with

Agreed, linters probably already help with that.

[asottile]

flake8 (via pyflakes) catches this for instance