BUG: Drastic memory usage increase in recent builds

[larsoner]

Describe the issue:

Recently in MNE-Python our CIs started dying due to memory overconsumption on both

• Linux: https://github.com/mne-tools/mne-python/runs/8253676336?check_suite_focus=true
• Windows: https://dev.azure.com/mne-tools/mne-python/_build/results?buildId=21722&view=logs&j=305851a9-a7bb-55db-0042-7e2b6f48aa1c&t=4d37777d-f36a-53aa-9217-6386d15dddcd&l=2570

On my local Linux machine doing the following as a test with git bisect --first-parent:

$ mprof run pytest -m "not ultraslowtest" mne/tests/ --tb=short --cov=mne --cov-report=xml --cov-report=html -vv && mprof plot

takes ~12GB of memory usage on main and also 45357de:

but only a much more reasonable ~2.5GB of max memory usage on db550d5:

So I think this is due to #21995.

MNE does use SciPy, Numba, scikit-learn, dipy, etc. that could be interacting with NumPy in some bad way, too. I could at least rule out numba, dipy, and scikit-learn by uninstalling them and seeing the same overconsumption, though (some tests will just be skipped or our library with just use slower code paths this way).

I'm happy to work on isolating this further if it's not clear why this might be happening just from having it pinned down to the PR/commit.

Reproduce the code example:

<not easily reproduced / not yet isolated>

Error message:

No response

NumPy/Python version information:

See commit numbers above

Context for the issue:

No response

[charris]

That's disturbing. Reverting is an option, but it would be nice to figure out why it has that effect.

[larsoner]

Tomorrow I'll see if I can replicate on something simpler like maybe just running numpy.test with memprof

[seberg]

Hmmmpf, the graph does look like we probably introduced a memory leak? Would be helpful to narrow it down a bit. Although if that is tricky, we can also run the debugging tools on the NumPy test suite to see if it points to something.

[rgommers]

SciPy CI jobs started dying as well with no indication of what the problem is other than Error: The operation was canceled. in GitHub Actions (example log). We should revert the PR, and then resubmit and investigate the problem.

Cc @eirrgang as author of the relevant PR.

[seberg]

oh, I missed that we know which PR is at fault. That should be an easy fix then, no need to revert.

[rgommers]

@seberg only if you already understand the problem and it can fixed within a few hours. Because it's going to be a huge waste of time if every downstream CI job testing numpy main is going to fail.

[rgommers]

Broken wheels have been deleted and memory leak should be fixed - see gh-22236.

[rgommers]

Thanks for finding the root cause @larsoner