Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Test with coverage freeze at a specific test file on Python 3.10.a7 #1151

Closed
fangchenli opened this issue Apr 28, 2021 · 8 comments
Closed

Test with coverage freeze at a specific test file on Python 3.10.a7 #1151

fangchenli opened this issue Apr 28, 2021 · 8 comments
Labels
bug Something isn't working not our bug The problem was elsewhere

Comments

@fangchenli
Copy link

Describe the bug
pandas started testing against Python 3.10.beta.1. pandas-dev/pandas#41113. While working on the CI, we found that running the coverage with pytest-cov would cause the test step to freeze immediately after start. But if use coveragepy directly, the test would freeze at a specific test file https://github.com/pandas-dev/pandas/blob/master/pandas/tests/io/json/test_pandas.py. For now, we skipped the whole file to make the CI green. This only happens on Python 3.10

To Reproduce

Build Version

INSTALLED VERSIONS
------------------
commit           : 1bdd5e6f668a4c5fabfa041082561340efe480df
python           : 3.10.0.beta.1
python-bits      : 64
OS               : Linux
OS-release       : 5.4.0-1046-azure
Version          : #48-Ubuntu SMP Tue Apr 13 07:18:42 UTC 2021
machine          : x86_64
processor        : x86_64
byteorder        : little
LC_ALL           : None
LANG             : C.UTF-8
LOCALE           : en_US.UTF-8

Output of pip list

Package          Version
---------------- ---------------------------
apipkg           1.5
attrs            20.3.0
coverage         5.6b2
Cython           0.29.23
execnet          1.8.0
hypothesis       6.10.1
iniconfig        1.1.1
numpy            1.21.0.dev0+1401.g1fee7acbc
packaging        20.9
pip              21.1
pluggy           0.13.1
py               1.10.0
pyparsing        2.4.7
pytest           6.3.0.dev399+g2049ae271
pytest-forked    1.3.0
pytest-xdist     2.2.1
python-dateutil  2.8.1
pytz             2021.1
setuptools       56.0.0
six              1.15.0
sortedcontainers 2.3.0
toml             0.10.2
wheel            0.36.2

Test command

coverage run -m pytest -m 'not slow and not network and not clipboard' pandas

Expected behavior
No freeze.
@fangchenli fangchenli added the bug Something isn't working label Apr 28, 2021
@nedbat
Copy link
Owner

nedbat commented Apr 28, 2021

Thanks for the report. Could you give me very specific instructions about how to run the tests?

I tried cloning the pandas repo, installing pytest and coverage. Then I ran your test command, and got:

$ coverage run -m pytest -m 'not slow and not network and not clipboard' pandas
ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --strict-data-files
  inifile: /private/tmp/pandas/pyproject.toml
  rootdir: /private/tmp/pandas

Coverage.py warning: No data was collected. (no-data-collected)

@nedbat
Copy link
Owner

nedbat commented Apr 29, 2021

BTW, where did you get a Python that claims to be 3.10.0.beta.1? That doesn't exist yet.

@fangchenli
Copy link
Author

Thanks for taking a look at this! @nedbat

Cloning the repo, and inside a venv with Python 3.10

# install dependencies
python -m pip install --upgrade pip setuptools wheel
pip install git+https://github.com/numpy/numpy.git
pip install git+https://github.com/pytest-dev/pytest.git
pip install git+https://github.com/nedbat/coveragepy.git
pip install cython python-dateutil pytz hypothesis pytest-xdist

# build pandas
python setup.py build_ext -q -j4
python -m pip install -e . --no-build-isolation --no-use-pep517

# run the problematic test with coverage
coverage run -m pytest -m 'not slow and not network and not clipboard' pandas/tests/io/json/test_pandas.py

GitHub Action pull the py310-dev from https://github.com/actions/python-versions/releases. And the latest version is 3.10.0-beta.1

@nedbat
Copy link
Owner

nedbat commented Apr 29, 2021

Just so you know, there is no 3.10 beta yet: actions/setup-python#207

@nedbat
Copy link
Owner

nedbat commented Apr 29, 2021

Thanks for the steps. Once I finally got numpy installed on my Mac, I ran your command and all the tests were skipped, because of the 3.10 skip in test_pandas.py. For others trying to reproduce this, it will help to create a branch in the pandas repo without that line.

When I commented out the skip, the tests ran, but crashed instead of hung:

 coverage run -m pytest -m 'not slow and not network and not clipboard' pandas/tests/io/json/test_pandas.py

======================================================================= test session starts =======================================================================
platform darwin -- Python 3.10.0a7, pytest-6.3.0.dev399+g2049ae271, py-1.10.0, pluggy-0.13.1
rootdir: /private/tmp/pandas, configfile: pyproject.toml
plugins: hypothesis-6.10.1, xdist-2.2.1, forked-1.3.0
collected 581 items

pandas/tests/io/json/test_pandas.py .........................................................................................................................x...x...........x...x...........x...x...........x...x............xx..xxxxxxxxxx..............................................xx..xx..xx..xx........................................................................................................................................................................................................................................Fatal Python error: Segmentation fault

Current thread 0x000000011948bdc0 (most recent call first):
  File "/private/tmp/pandas/pandas/tests/io/json/test_pandas.py", line 1197 in test_tz_is_utc
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/python.py", line 185 in pytest_pyfunc_call
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/python.py", line 1652 in runtest
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 261 in <lambda>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 340 in from_call
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 260 in call_runtest_hook
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 221 in call_and_report
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 132 in runtestprotocol
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/main.py", line 352 in pytest_runtestloop
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/main.py", line 327 in _main
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/main.py", line 273 in wrap_session
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/main.py", line 320 in pytest_cmdline_main
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/config/__init__.py", line 165 in main
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/_pytest/config/__init__.py", line 188 in console_main
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/pytest/__main__.py", line 5 in <module>
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/coverage/execfile.py", line 247 in run
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/coverage/cmdline.py", line 746 in do_run
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/coverage/cmdline.py", line 588 in command_line
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/lib/python3.10/site-packages/coverage/cmdline.py", line 871 in main
  File "/usr/local/virtualenvs/tmp-38009ba9b861cca/bin/coverage", line 8 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg.lapack_lite, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.strptime, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.tslib, pandas._libs.lib, pandas._libs.hashing, pandas._libs.ops, pandas._libs.arrays, pandas._libs.index, pandas._libs.join, pandas._libs.sparse, pandas._libs.reduction, pandas._libs.indexing, pandas._libs.internals, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.parsers, pandas._libs.json, pandas._libs.testing, _scproxy (total: 56)
zsh: segmentation fault  coverage run -m pytest -m 'not slow and not network and not clipboard'

Any idea why?

@fangchenli fangchenli changed the title Test with coverage freeze at a specific test file on Python 3.10.beta.1 Test with coverage freeze at a specific test file on Python 3.10.a7 Apr 29, 2021
@nedbat
Copy link
Owner

nedbat commented Jul 13, 2021

Has this improved with the later 3.10 betas?

@fangchenli
Copy link
Author

fangchenli commented Jul 13, 2021 •
edited

Has this improved with the later 3.10 betas?

The root cause of this problem is a segfault from pandas' json extension on Python 3.10 (pandas-dev/pandas#42130). When testing locally, we can see the segfault directly. But it somehow froze the CI when running with coverage.

Right now that segfault is magically disappeared. We haven't looked into it yet.

@nedbat
Copy link
Owner

nedbat commented Oct 18, 2021

It sounds like pandas fixed an issue? Feel free to reopen this if it comes back.

@nedbat nedbat closed this as completed Oct 18, 2021
@nedbat nedbat added the not our bug The problem was elsewhere label Oct 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working not our bug The problem was elsewhere
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@nedbat @fangchenli