Enable memray for tests.

[WilliamJamieson]

Description

The PR adds the pytest-memray plugin which is supposed to help us track memory usage in our test suite. This work is in anticipation of bloomberg/pytest-memray#52, which should allow us to detect some memory leaks in astropy which would be exposed by our test suite. I think this might be useful considering all the C code used by astropy.

Checklist for package maintainer(s)

This checklist is meant to remind the package maintainer(s) who will review this pull request of some common things to look for. This list is not exhaustive.

• Do the proposed changes actually accomplish desired goals?
• Do the proposed changes follow the Astropy coding guidelines?
• Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
• Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
• Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see "When to rebase and squash commits".
• Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the Extra CI label. Codestyle issues can be fixed by the bot.
• Is a change log needed? If yes, did the change log check pass? If no, add the no-changelog-entry-needed label. If this is a manual backport, use the skip-changelog-checks label unless special changelog handling is necessary.
• Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
• Is a milestone set? Milestone must be set but astropy-bot check might be missing; do not let the green checkmark fool you.
• At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate backport-X.Y.x label(s) before merge.

[github-actions]

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

[pllim]

Do we need memray in so many jobs?

[WilliamJamieson]

I suppose not, but if we want to start decorating tests for memray to enforce memory limits (and maybe the leak checks), this will require --memray to give them actual force.

[pllim]

This is what I see in the log:

================================ MEMRAY REPORT =================================
Allocations results for .tox/py310-test-cov/lib/python3.10/site-packages/astropy/convolution/tests/test_convolve_fft.py::TestConvolve2D::test_big_fail

	 📦 Total memory allocated: 2.0GiB
	 📏 Total allocations: 8
	 📊 Histogram of allocation sizes: |█    |
	 🥇 Biggest allocating functions:
		- test_big_fail:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/convolution/tests/test_convolve_fft.py:794 -> 2.0GiB
		- convolve_fft:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/convolution/convolve.py:740 -> 32.0KiB
		- __init__:/opt/hostedtoolcache/Python/3.10.8/x64/lib/python3.10/inspect.py:2965 -> 576.0B


Allocations results for .tox/py310-test-cov/lib/python3.10/site-packages/astropy/convolution/tests/test_convolve_fft.py::TestConvolve2D::test_padding

	 📦 Total memory allocated: 240.9MiB
	 📏 Total allocations: 10691033
	 📊 Histogram of allocation sizes: |█ ▁     ▇|
	 🥇 Biggest allocating functions:
		- _raw_fft:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/fft/_pocketfft.py:73 -> 20.6MiB
		- _raw_fft:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/fft/_pocketfft.py:73 -> 20.6MiB
		- _raw_fft:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/fft/_pocketfft.py:73 -> 20.6MiB
		- _raw_fft:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/fft/_pocketfft.py:73 -> 20.6MiB
		- convolve_fft:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/convolution/convolve.py:911 -> 20.6MiB


Allocations results for .tox/py310-test-cov/lib/python3.10/site-packages/astropy/visualization/tests/test_histogram.py::test_histogram_pathological_input

	 📦 Total memory allocated: 200.9MiB
	 📏 Total allocations: 49
	 📊 Histogram of allocation sizes: |█       ▁|
	 🥇 Biggest allocating functions:
		- freedman_bin_width:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/stats/histogram.py:267 -> 200.9MiB
		- _unique1d:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/lib/arraysetops.py:354 -> 48.0B
		- _lerp:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/lib/function_base.py:4530 -> 32.0B
		- _unique1d:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/lib/arraysetops.py:352 -> 16.0B
		- _get_indexes:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/lib/function_base.py:4631 -> 16.0B


Allocations results for .tox/py310-test-cov/lib/python3.10/site-packages/astropy/nddata/tests/test_ccddata.py::test_read_wcs_not_creatable

	 📦 Total memory allocated: 71.6MiB
	 📏 Total allocations: 5209
	 📊 Histogram of allocation sizes: |█ ▂      |
	 🥇 Biggest allocating functions:
		- readarray:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/fits/file.py:355 -> 35.7MiB
		- ones:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/numpy/core/numeric.py:204 -> 35.7MiB
		- _verify:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/fits/card.py:1230 -> 39.9KiB
		- __init__:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/fits/card.py:189 -> 14.1KiB
		- __init__:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/fits/card.py:189 -> 14.1KiB


Allocations results for .tox/py310-test-cov/lib/python3.10/site-packages/astropy/utils/iers/tests/test_iers.py::TestBasic::test_simple[IERS]

	 📦 Total memory allocated: 36.6MiB
	 📏 Total allocations: 2709
	 📊 Histogram of allocation sizes: |█ ▁ ▃ ▁ █|
	 🥇 Biggest allocating functions:
		- <listcomp>:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/ascii/fixedwidth.py:39 -> 21.0MiB
		- get_lines:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/ascii/core.py:338 -> 4.0MiB
		- read:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/ascii/core.py:1454 -> 3.0MiB
		- __new__:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/table/column.py:557 -> 2.7MiB
		- generic_converter:/home/runner/work/astropy/astropy/.tox/py310-test-cov/lib/python3.10/site-packages/astropy/io/ascii/core.py:1044 -> 2.7MiB

tl;dr -- I guess it only reports the top 5 offenders or something?

This is a lot of info to digest and I am afraid that all this CPU cycles will be wasted if people ignore it most of the time anyway. I wonder if it should be very targeted like the timing benchmarks. But since even that is broken right now, I dunno. But definitely not to be run every push. Maybe when we can agree on what to actually profile, we can move it to a cron job and it can be activated on-demand for PR by a special label.

[WilliamJamieson]

@pllim, I agree that the report would be better used as part of the benchmarks and/or as part of a cron. The number of items reported is controllable via the --most-allocations argument to pytest, see https://pytest-memray.readthedocs.io/en/latest/configuration.html). Which should help us produce a more usable report.

There are two use cases that I can see memray useful for in astropy:

1. Enabling us to decorate unit tests with memory limits, so that we can make sure changes aren't causing spikes in memory usage in particularly memory sensitive areas.
2. (with a forthcoming feature) Enable us to decorate tests to help us catch memory leaks and/or show that such leaks remain fixed.

Note that if pytest-memray is installed and --memray is not an argument pytest-memray will be run on any test decorated with a pytest-memray decorator. Which means we don't have to run --memray in general for all tests.

Obviously, there are no such decorators for tests included in this PR as I don't really know where to start with where to place those; however, the main part of this PR is to demonstrate that we can run pytest-memray on astropy. Which is now possible since bloomberg/pytest-memray#60 was fixed (and released as part of version 1.4.0).

The other part of this PR was to start getting a baseline for memory usage across astropy (to figure out where to target memory usage/leak tests), which as suggested maybe better suited to a cron job for now.

[saimn]

I agree we don't need to run it in CI for all tests like this, though there may be some tests where we can decrease the size of the arrays. But i have no idea, what is the overhead with memray ?
What could be useful imo would be to test the main data structures of Astropy, i.e. Table, NDData, Quantity, Time, SkyCoord, with small loops and the memray decorator to check if there are some memory leaks.