Skip to content

Commit

Permalink
python: skip pytest_pycollect_makeitem work on certain names
Browse files Browse the repository at this point in the history
When a Python object (module/class/instance) is collected, for each name
in `obj.__dict__` (and up its MRO) the pytest_pycollect_makeitem hook is
called for potentially creating a node for it.

These Python objects have a bunch of builtin attributes that are
extremely unlikely to be collected. But due to their pervasiveness,
dispatching the hook for them ends up being mildly expensive and also
pollutes PYTEST_DEBUG=1 output and such.

Let's just ignore these attributes.

On the pandas test suite commit 04e9e0afd476b1b8bed930e47bf60e,
collect only, irrelevant lines snipped, about 5% improvement:

Before:

```
         51195095 function calls (48844352 primitive calls) in 39.089 seconds

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
226602/54    0.145    0.000   38.940    0.721 manager.py:90(_hookexec)
    72227    0.285    0.000   20.146    0.000 python.py:424(_makeitem)
    72227    0.171    0.000   16.678    0.000 python.py:218(pytest_pycollect_makeitem)
```

After:

```
          48410921 function calls (46240870 primitive calls) in 36.950 seconds

    ncalls  tottime  percall  cumtime  percall filename:lineno(function)
 181429/54    0.113    0.000   36.777    0.681 manager.py:90(_hookexec)
     27054    0.130    0.000   17.755    0.001 python.py:465(_makeitem)
     27054    0.121    0.000   16.219    0.001 python.py:218(pytest_pycollect_makeitem)
```
  • Loading branch information
bluetech committed Aug 24, 2020
1 parent 143e3ab commit 4f1233e
Showing 1 changed file with 23 additions and 0 deletions.
23 changes: 23 additions & 0 deletions src/_pytest/python.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import itertools
import os
import sys
import types
import typing
import warnings
from collections import Counter
Expand Down Expand Up @@ -341,6 +342,26 @@ def reportinfo(self) -> Tuple[Union[py.path.local, str], int, str]:
return fspath, lineno, modpath


# As an optimization, these builtin attribute names are pre-ignored when
# iterating over an object during collection -- the pytest_pycollect_makeitem
# hook is not called for them.
# fmt: off
class _EmptyClass: pass # noqa: E701
IGNORED_ATTRIBUTES = frozenset.union( # noqa: E305
frozenset(),
# Module.
dir(types.ModuleType("empty_module")),
# Some extra module attributes the above doesn't catch.
{"__builtins__", "__file__", "__cached__"},
# Class.
dir(_EmptyClass),
# Instance.
dir(_EmptyClass()),
)
del _EmptyClass
# fmt: on


class PyCollector(PyobjMixin, nodes.Collector):
def funcnamefilter(self, name: str) -> bool:
return self._matches_prefix_or_glob_option("python_functions", name)
Expand Down Expand Up @@ -402,6 +423,8 @@ def collect(self) -> Iterable[Union[nodes.Item, nodes.Collector]]:
# Note: seems like the dict can change during iteration -
# be careful not to remove the list() without consideration.
for name, obj in list(dic.items()):
if name in IGNORED_ATTRIBUTES:
continue
if name in seen:
continue
seen.add(name)
Expand Down

0 comments on commit 4f1233e

Please sign in to comment.