Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expand warnings output for ResourceWarning #9682

Merged
merged 3 commits into from
Feb 15, 2022
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
4 changes: 4 additions & 0 deletions changelog/9644.improvement.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
More information about the location of resources that led Python to raise :class:`ResourceWarning` can now
be obtained by enabling :mod:`tracemalloc`.

See :ref:`resource-warnings` for more information.
15 changes: 15 additions & 0 deletions doc/en/how-to/capture-warnings.rst
Original file line number Diff line number Diff line change
Expand Up @@ -441,3 +441,18 @@ Please read our :ref:`backwards-compatibility` to learn how we proceed about dep
features.

The full list of warnings is listed in :ref:`the reference documentation <warnings ref>`.


.. _`resource-warnings`:

Resource Warnings
-----------------

Additional information of the source of a :class:`ResourceWarning` can be obtained when captured by pytest if
:mod:`tracemalloc` module is enabled.

One convenient way to enable :mod:`tracemalloc` when running tests is to set the :envvar:`PYTHONTRACEMALLOC` to a large
enough number of frames (say ``20``, but that number is application dependent).

For more information, consult the `Python Development Mode <https://docs.python.org/3/library/devmode.html>`__
section in the Python documentation.
19 changes: 19 additions & 0 deletions src/_pytest/warnings.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,25 @@ def warning_record_to_str(warning_message: warnings.WarningMessage) -> str:
warning_message.lineno,
warning_message.line,
)
if warning_message.source is not None:
try:
import tracemalloc
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there platforms which don't have tracemalloc available?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pypy3 CI failed with ImportError: cannot import _tracemalloc, so I went ahead and assumed that was the case, but didn't look into official sources TBH.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is likely that PyPy doesn't support it at all due to differing memory management.

except ImportError:
pass
else:
tb = tracemalloc.get_object_traceback(warning_message.source)
if tb is not None:
formatted_tb = "\n".join(tb.format())
# Use a leading new line to better separate the (large) output
# from the traceback to the previous warning text.
msg += (
f"\nObject allocated at (most recent call first):\n{formatted_tb}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure most recent call first is correct here, maybe just remove it?

)
else:
# No need for a leading new line.
url = "https://docs.pytest.org/en/stable/how-to/capture-warnings.html#resource-warnings"
msg += "Enable tracemalloc to get traceback where the object was allocated.\n"
msg += f"See {url} for more info."
return msg


Expand Down
51 changes: 51 additions & 0 deletions testing/test_warnings.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import os
import sys
import warnings
from typing import List
from typing import Optional
Expand Down Expand Up @@ -774,3 +775,53 @@ def test_it():
"*Unknown pytest.mark.unknown*",
]
)


def test_resource_warning(pytester: Pytester, monkeypatch: pytest.MonkeyPatch) -> None:
# Some platforms (notably PyPy) don't have tracemalloc.
# We choose to explicitly not skip this in case tracemalloc is not
# available, using `importorskip("tracemalloc")` for example,
# because we want to ensure the same code path does not break in those platforms.
try:
import tracemalloc # noqa

has_tracemalloc = True
except ImportError:
has_tracemalloc = False

pytester.makepyfile(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this test (tracemalloc not enabled) will fail if the person running the pytest test suite enables tracemalloc.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm why do you say that? And do you mean running in CPython?

I tried it here:

python -X tracemalloc=20 -m pytest testing\test_warnings.py

And it passes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note the try/except is checking if tracemalloc is available at all, which is not on PyPy for example, not if tracemalloc is enabled or not.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess python -X tracemalloc=20 doesn't transfer, but an envvar does, so try this:

PYTHONTRACEMALLOC=100 pytest testing/test_warnings.py -k resource

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh got it, good catch. 😁

Updated the test, thanks.

"""
def open_file(p):
f = p.open("r")
assert p.read_text() == "hello"

def test_resource_warning(tmp_path):
p = tmp_path.joinpath("foo.txt")
p.write_text("hello")
open_file(p)
"""
)
result = pytester.run(sys.executable, "-Xdev", "-m", "pytest")
expected_extra = (
[
"*ResourceWarning* unclosed file*",
"*Enable tracemalloc to get traceback where the object was allocated*",
"*See https* for more info.",
]
if has_tracemalloc
else []
)
result.stdout.fnmatch_lines([*expected_extra, "*1 passed*"])

monkeypatch.setenv("PYTHONTRACEMALLOC", "20")

result = pytester.run(sys.executable, "-Xdev", "-m", "pytest")
expected_extra = (
[
"*ResourceWarning* unclosed file*",
"*Object allocated at (most recent call first)*",
]
if has_tracemalloc
else []
)
result.stdout.fnmatch_lines([*expected_extra, "*1 passed*"])