AttributeError: module 'numpy' has no attribute 'bool' in scheduled CI tests

[joshuacwnewton]

Failing tests: https://github.com/spinalcordtoolbox/spinalcordtoolbox/actions/runs/3730836462/attempts/1

Error messages:

Check if voxelmorph is installed....................[FAIL]
An error occured while importing module voxelmorph -> module 'numpy' has no attribute 'bool'
Full traceback: Traceback (most recent call last):
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/spinalcordtoolbox/scripts/sct_check_dependencies.py", line 299, in main
    module = module_import(module_name, suppress_stderr)
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/spinalcordtoolbox/scripts/sct_check_dependencies.py", line 105, in module_import
    module = importlib.import_module(module_name)
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 843, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/voxelmorph/__init__.py", line 12, in <module>
    import neurite
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/neurite/__init__.py", line 19, in <module>
    import pystrum
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/pystrum/__init__.py", line 3, in <module>
    from . import pynd
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/pystrum/pynd/__init__.py", line 1, in <module>
    from . import ndutils
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/pystrum/pynd/ndutils.py", line 581, in <module>
    def sphere_vol(vol_shape, radius, center=None, dtype=np.bool):
  File "/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/numpy/__init__.py", line 284, in __getattr__
    raise AttributeError("module {!r} has no attribute "
AttributeError: module 'numpy' has no attribute 'bool'

Check if wquantiles is installed....................[OK] (0.4)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
Check PropSeg compatibility with OS ................[OK]
Check if figure can be opened with PyQt.............[FAIL] ($DISPLAY not set on X11-supporting system)
/home/runner/work/spinalcordtoolbox/spinalcordtoolbox/python/envs/venv_sct/lib/python3.8/site-packages/pystrum/pynd/ndutils.py:581: FutureWarning: In the future `np.bool` will be defined as the corresponding NumPy scalar.  (This may have returned Python scalars in past versions.
Check if figure can be opened with matplotlib.......[FAIL] (Using non-GUI backend 'agg')
  def sphere_vol(vol_shape, radius, center=None, dtype=np.bool):

[joshuacwnewton]

This error is due to the recent release of Numpy 1.24, which included several new deprecations: https://github.com/numpy/numpy/releases/tag/v1.24.0

The deprecation for the aliases np.object, np.bool, np.float,
np.complex, np.str, and np.int is expired (introduces NumPy
1.20). Some of these will now give a FutureWarning in addition to
raising an error since they will be mapped to the NumPy scalars in
the future. (numpy/numpy#22607)

This numpy update conflicts with pystrum, which is installed via voxelmorph:

(venv_sct) joshua@tadpole:~/repos/spinalcordtoolbox$ pipdeptree -r -p pystrum
pystrum==0.2
  - neurite==0.2 [requires: pystrum>=0.2]
    - voxelmorph==0.2 [requires: neurite>=0.2]

Our version of pystrum is the latest version (pystrum==0.2, June 2022), and the package's source code on master hasn't been updated since that release. Meaning that the deprecation of np.bool has gone unaddressed upstream:

https://github.com/adalca/pystrum/blob/8cd5c483195971c0c51e9809f33aa04777aa35c8/pystrum/pynd/ndutils.py#L581

[joshuacwnewton]

Given that pystrum is (infrequently) maintained, with 0 open issues, I decided to check one level above pystrum @ neurite, given that that package is just as susceptible to this issue.

However, neurite has the same maintainer as pystrum, so either way, we're going to be talking to the same person. 😓

[joshuacwnewton]

In the short term, we can at least pin numpy<1.24, and report the issue upstream.

In the long term, hopefully pystrum gets an update!

(Though, I'm curious how numpy==1.24 will affect SCT's test suite, too, since the error in sct_check_dependencies occurs before any tests can be run. I have a feeling this isn't the only deprecation that will cause issues, given how poor a track record we have of keeping up with FutureWarnings and etc.)

[joshuacwnewton]

Curiously, the "Windows (Native)" CI run passed (while all others failed).

In that case, however, numpy=1.21.6 is installed (rather than numpy==1.24). I'm not sure why this is; I'd have to run pipdeptree on a fresh Windows installation...

(It is a little concerning that our versions of numpy aren't consistent across OSs, though.)

[kousu]

I just tried installing 5.7 to see if this would bite but it's okay

p115628@joplin:~$ curl -JLO https://github.com/spinalcordtoolbox/spinalcordtoolbox/archive/refs/tags/5.7.tar.gz 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 2762k    0 2762k    0     0  1759k      0 --:--:--  0:00:01 --:--:-- 5421k
p115628@joplin:~$ tar -zxvf spinalcordtoolbox-5.7.tar.gz spinalcordtoolbox-5.7/
p115628@joplin:~$ cd spinalcordtoolbox-5.7/
p115628@joplin:~/spinalcordtoolbox-5.7$ ./install_sct -y
[...]
--
Spinal Cord Toolbox (5.7)

sct_check_dependencies 
--


SYSTEM INFORMATION
------------------
SCT info:
- version: 5.7
- path: /home/GRAMES.POLYMTL.CA/p115628/sct_5.7
OS: linux (Linux-5.15.0-56-generic-x86_64-with-debian-bookworm-sid)
CPU cores: Available: 128, Used by ITK functions: 128
RAM: Total: 257591MB, Used: 2394MB, Available: 235486MB

OPTIONAL DEPENDENCIES
---------------------
Check FSLeyes version...............................[OK] (1.5.0)

MANDATORY DEPENDENCIES
----------------------
Check Python executable.............................[OK]
  Using bundled python 3.7.15 (default, Nov 24 2022, 21:12:53) 
[GCC 11.2.0] at /home/GRAMES.POLYMTL.CA/p115628/sct_5.7/python/envs/venv_sct/bin/python
Check if data are installed.........................[OK]
Check if dipy is installed..........................[OK] (1.5.0)
Check if ivadomed is installed......................[OK] (2.9.6)
Check if matplotlib is installed....................[OK] (3.5.2)
Check if nibabel is installed.......................[OK] (3.2.2)
Check if nilearn is installed.......................[OK] (0.9.1)
Check if numpy is installed.........................[OK] (1.21.6)
Check if onnxruntime is installed...................[OK] (1.7.0)
Check if pandas is installed........................[OK] (1.3.5)
Check if portalocker is installed...................[OK] (2.5.1)
Check if psutil is installed........................[OK] (5.9.1)
Check if pyqt5 (5.11.3) is installed................[OK] (5.11.3)
Check if pytest is installed........................[OK] (7.1.2)
Check if pytest-cov is installed....................[OK] (3.0.0)
Check if raven is installed.........................[OK]
Check if requests is installed......................[OK] (2.28.1)
Check if requirements-parser is installed...........[OK]
Check if scipy is installed.........................[OK] (1.7.3)
Check if scikit-image is installed..................[OK] (0.19.3)
Check if scikit-learn is installed..................[OK] (1.0.2)
Check if xlwt is installed..........................[OK] (1.3.0)
Check if tqdm is installed..........................[OK] (4.64.0)
Check if transforms3d is installed..................[OK] (0.3.1)
Check if urllib3 is installed.......................[OK] (1.26.11)
Check if pytest_console_scripts is installed........[OK]
Check if pyyaml is installed........................[OK] (6.0)
Check if voxelmorph is installed....................[OK] (0.2)
Check if wquantiles is installed....................[OK] (0.4)
Check if spinalcordtoolbox is installed.............[OK]
Check ANTs compatibility with OS ...................[OK]
[OK]
Check if figure can be opened with matplotlib.......[OK] (Using GUI backend: 'QtAgg')
Check if figure can be opened with PyQt.............libGL error: MESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: Ne peut ouvrir le fichier d'objet partagé: Aucun fichier ou dossier de ce type (search paths /usr/lib/x86_64-linux-gnu/dri:\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)
libGL error: failed to load driver: swrast
[OK]

which is because requirements-freeze.txt saved us (this time (, Batman!)).