Backport PR #11962: Fix warnings with Python 3.10 #12159

saimn · 2021-09-10T06:36:32Z

Description

Backport PR #11962

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

Do the proposed changes actually accomplish desired goals?
Do the proposed changes follow the Astropy coding guidelines?
Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label.
Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label. If this is a manual backport, use the skip-changelog-checks label unless special changelog handling is necessary.
Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

pllim · 2021-09-10T12:54:54Z

Why does this look familiar, @eslavich ?

E       asdf.exceptions.AsdfWarning: asdf.resource_mappings plugin from package asdf==2.9.0.dev11+g9e64fbe failed to load:
E       
E       RuntimeError: Unable to locate core schemas

eslavich · 2021-09-10T13:02:39Z

G-g-g-g-g-g-ghosts!

eslavich · 2021-09-10T13:05:02Z

Hmm, that was a failure in the original PR that apparently went away after I ignored it long enough:

#11962 (comment)

I don't suppose that strategy will prove effective here...

pllim · 2021-09-10T13:06:58Z

Is there another PR need backporting or what? 🤷

saimn · 2021-10-14T20:42:29Z

Tests with asdf are still failing...
Also some failures that will be fixed by #12253.

pllim · 2021-10-14T21:02:55Z

@eslavich , this doesn't ring any bell at all?

asdf.exceptions.AsdfWarning: asdf.resource_mappings plugin from package asdf==2.9.0.dev14+gee34d21 failed to load:

RuntimeError: Unable to locate core schemas

pllim · 2021-10-26T13:30:45Z

Maybe this isn't as critical if we'll never do another v4.3.x release... Still would be nice to have closure, @eslavich .

eslavich · 2021-11-03T21:14:18Z

I spent some time today trying to reproduce this on my (macOS) laptop, no success so far. The failing tests all pass with Python 3.10, asdf master, and astropy 4.3.1.

pllim · 2021-11-10T16:40:29Z

@eslavich , did you turn warnings into failures like we do?

pllim · 2021-11-10T16:40:51Z

p.s. You should maybe try with this branch, not the released 4.3.1.

eslavich · 2021-11-10T16:46:42Z

Unfortunately I'm not currently able to install astropy from source with Python 3.10 due to a compiler issue that I haven't had the time to troubleshoot. This is the error in case someone has already slain this particular dragon:

  ./astropy/stats/_stats.c:23373:5: error: expression is not assignable
      --Py_REFCNT(o);

pllim · 2021-11-10T16:47:52Z

Hmm... stats... Maybe @larrybradley or @dhomeier know?

larrybradley · 2021-11-10T17:17:43Z

I have no clue about that error. There are Python 3.10 wheels for 5.0rc1: https://pypi.org/project/astropy/5.0rc1/#files

dhomeier · 2021-11-10T20:40:22Z

Can't really say what's going wrong there, but I don't recall ever seeing it in my 3.10 builds.
What compiler is that, and which Cython version was the file built with (my _stats.c was last cythoned 22 October; maybe a rebuild will help)?
It has a #define around L500 that looks like it should take care of this for 3.9 and above:

#if PY_VERSION_HEX >= 0x030900A4
  #define __Pyx_SET_REFCNT(obj, refcnt) Py_SET_REFCNT(obj, refcnt)
  #define __Pyx_SET_SIZE(obj, size) Py_SET_SIZE(obj, size)
#else
  #define __Pyx_SET_REFCNT(obj, refcnt) Py_REFCNT(obj) = (refcnt)
  #define __Pyx_SET_SIZE(obj, size) Py_SIZE(obj) = (size)
#endif

eslavich · 2021-11-10T20:48:56Z

This is the compiler it's using:

$ clang --version
Apple clang version 12.0.0 (clang-1200.0.32.2)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

eslavich · 2021-11-10T21:36:27Z

I tried fiddling with the compiler versions, to no avail, but then I deleted the _stats.c file and that got me through the install (it didn't reappear, but its absence doesn't cause a problem for the asdf tests).

The tests are still passing on my laptop with:

Python 3.10
astropy backport-11962 (this PR's branch)
asdf master

It is turning warnings into errors -- I plunked down a warning into one of the tests and got it to fail.

dhomeier · 2021-11-10T21:49:27Z

Well, that's a much more up-to-date Xcode than my Mojave installation.

I tried fiddling with the compiler versions, to no avail, but then I deleted the _stats.c file and that got me through the install (it didn't reappear, but its absence doesn't cause a problem for the asdf tests).

😮 No cythoning ./astropy/stats/_stats.pyx to ./astropy/stats/_stats.c when running setup.py build?
Do you have a _stats.cpython-310-darwin.so in that subdirectory or the respective one in build/lib.macosc*-3.10?
And does astropy.test(package='stats', args='-vk _kuiper_') pass?

dhomeier · 2021-11-10T22:30:36Z

For this PR pulled into 4.3.x, I don't see any failures on testing io (built with ASTROPY_USE_SYSTEM_CFITSIO=0 since the patches for the system cfitsio 4.0 are not in here; have not set filterwarnings=error), but the warning above does appear, and an io.fits deprecation (along with a bunch of "unclosed file"s):

lib/python3.10/site-packages/astropy/io/fits/tests/test_table.py::TestVLATables::test_variable_length_table_format_pd_from_list
lib/python3.10/site-packages/astropy/io/fits/tests/test_table.py::TestVLATables::test_copy_vla
lib/python3.10/site-packages/astropy/io/fits/tests/test_table.py::TestColumnFunctions::test_p_column_deepcopy
  /Users/derek/lib/python3.10/site-packages/astropy/io/fits/column.py:649: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
    array = np.array(array)

lib/python3.10/site-packages/astropy/io/fits/tests/test_table.py::TestVLATables::test_vla_with_gap
  /opt/lib/python3.10/site-packages/_pytest/python.py:183: ResourceWarning: unclosed file <_io.FileIO name='/Users/derek/lib/python3.10/site-packages/astropy/io/fits/tests/data/theap-gap.fits' mode='rb' closefd=True>
    result = testfunction(**testargs)

lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_angle.py::test_angle
  /opt/lib/python3.10/site-packages/asdf/entry_points.py:44: AsdfWarning: asdf.resource_mappings plugin from package asdf==2.8.1 failed to load:
  
  RuntimeError: Unable to locate core schemas
    warnings.warn(

lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_angle.py: 9 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_earthlocation.py: 18 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_frames.py: 15 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_representation.py: 42 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_skycoord.py: 34 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/coordinates/tests/test_spectralcoord.py: 9 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/fits/tests/test_fits.py: 7 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/table/tests/test_table.py: 36 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/time/tests/test_time.py: 15 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/time/tests/test_timedelta.py: 36 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/transform/tests/test_transform.py: 534 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/transform/tests/test_units_mapping.py: 18 warnings
lib/python3.10/site-packages/astropy/io/misc/asdf/tags/unit/tests/test_equivalency.py: 75 warnings
  /opt/lib/python3.10/site-packages/asdf/tests/httpserver.py:44: DeprecationWarning: isSet() is deprecated, use is_set() instead
    while not stop_event.isSet():

eslavich · 2021-11-10T22:51:28Z

Ah ha! Thanks @dhomeier, I had asdf installed in editable mode and that's the difference. I can reproduce this now.

eslavich · 2021-11-11T15:59:32Z

The ASDF problem is due to an issue in pytest that has been fixed in main but not yet released:

pytest-dev/pytest#9169

It goes away when dev pytest is installed.

pllim · 2021-11-11T16:06:53Z

The ASDF problem is due to an issue in pytest that has been fixed in main but not yet released

I still don't understand why it is only a problem in this PR but not in main here.

eslavich · 2021-11-11T16:15:33Z

I still don't understand why it is only a problem in this PR but not in main here.

A load order thing I wonder? The problem occurs because pytest replaces one of the importlib.resources objects with an assertion rewriting proxy that doesn't implement a new 3.10 method. If asdf is able to load the entry point before that gets subbed in, there isn't an issue. We did see this failure in the original PR but then it disappeared after the branch was updated.

pllim · 2021-11-11T16:24:27Z

@saimn , so maybe we should just ignore that warning here? Is that possible?

pllim · 2022-01-11T15:50:14Z

Do we still need this? I don't think we will release 4.x anymore.

saimn · 2022-01-11T15:56:54Z

Yep, closing.

saimn added this to the v4.3.2 milestone Sep 10, 2021

github-actions bot added convolution io.fits testing utils labels Sep 10, 2021

saimn added the skip-basebranch-check Skip base branch check for direct backports label Sep 10, 2021

saimn closed this Sep 10, 2021

saimn reopened this Sep 10, 2021

saimn added the skip-changelog-checks Tells bot to skip changlog checks label Sep 10, 2021

Backport PR astropy#11962: Fix warnings with Python 3.10

dea513d

saimn force-pushed the backport-11962 branch from 1627ddd to dea513d Compare October 14, 2021 20:21

saimn closed this Jan 11, 2022

saimn deleted the backport-11962 branch January 11, 2022 15:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Backport PR #11962: Fix warnings with Python 3.10 #12159

Backport PR #11962: Fix warnings with Python 3.10 #12159

saimn commented Sep 10, 2021 •

edited

pllim commented Sep 10, 2021

eslavich commented Sep 10, 2021

eslavich commented Sep 10, 2021

pllim commented Sep 10, 2021

saimn commented Oct 14, 2021

pllim commented Oct 14, 2021

pllim commented Oct 26, 2021

eslavich commented Nov 3, 2021

pllim commented Nov 10, 2021

pllim commented Nov 10, 2021

eslavich commented Nov 10, 2021

pllim commented Nov 10, 2021

larrybradley commented Nov 10, 2021

dhomeier commented Nov 10, 2021

eslavich commented Nov 10, 2021

eslavich commented Nov 10, 2021

dhomeier commented Nov 10, 2021 •

edited

dhomeier commented Nov 10, 2021

eslavich commented Nov 10, 2021

eslavich commented Nov 11, 2021

pllim commented Nov 11, 2021

eslavich commented Nov 11, 2021

pllim commented Nov 11, 2021

pllim commented Jan 11, 2022

saimn commented Jan 11, 2022

Backport PR #11962: Fix warnings with Python 3.10 #12159

Backport PR #11962: Fix warnings with Python 3.10 #12159

Conversation

saimn commented Sep 10, 2021 • edited

Description

Checklist for package maintainer(s)

pllim commented Sep 10, 2021

eslavich commented Sep 10, 2021

eslavich commented Sep 10, 2021

pllim commented Sep 10, 2021

saimn commented Oct 14, 2021

pllim commented Oct 14, 2021

pllim commented Oct 26, 2021

eslavich commented Nov 3, 2021

pllim commented Nov 10, 2021

pllim commented Nov 10, 2021

eslavich commented Nov 10, 2021

pllim commented Nov 10, 2021

larrybradley commented Nov 10, 2021

dhomeier commented Nov 10, 2021

eslavich commented Nov 10, 2021

eslavich commented Nov 10, 2021

dhomeier commented Nov 10, 2021 • edited

dhomeier commented Nov 10, 2021

eslavich commented Nov 10, 2021

eslavich commented Nov 11, 2021

pllim commented Nov 11, 2021

eslavich commented Nov 11, 2021

pllim commented Nov 11, 2021

pllim commented Jan 11, 2022

saimn commented Jan 11, 2022

saimn commented Sep 10, 2021 •

edited

dhomeier commented Nov 10, 2021 •

edited