Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ecosystem compatibility with numpy 2.0 #26191

Open
rgommers opened this issue Apr 1, 2024 · 72 comments
Open

Ecosystem compatibility with numpy 2.0 #26191

rgommers opened this issue Apr 1, 2024 · 72 comments

Comments

@rgommers
Copy link
Member

rgommers commented Apr 1, 2024

This list tracks the compatibility status of packages that depend on or support NumPy. If "compatible release on PyPI" does not say "yes" but a version number is listed: this is based on plans announced in a tracking issue or other communication by the authors of the package.

Maintainers: please feel free to edit directly (please refresh the page first to avoid overwriting edits from others!). Others who want to update things: please do comment, or feel free to ping me elsewhere.

Package name Compatible release on PyPI? Min compatible version Notes
AstroPy yes 6.1.0 astropy/astropy#16200
AwkwardArray yes 2.6.3 scikit-hep/awkward#3064
BioPython 1.84 biopython/biopython#4676
Bokeh yes 3.4.1 bokeh/bokeh#13835
Boost.Python 1.86.0 boostorg/python#431
Cartopy yes 0.23 SciTools/cartopy#2339
ContourPy yes 1.2.1 contourpy/contourpy#371
CuPy 14.0.0 cupy/cupy#8306
Cython yes 3.0.4 Version is an estimate, it's worked fine for quite a while
Dask yes 2024.5.1 dask/dask#11066
GDAL yes 3.9.0 OSGeo/gdal#9751
GeoPandas yes 0.14.4 geopandas/geopandas#3258
h5py yes 3.11.0 h5py/h5py#2353
hypothesis yes 6.100.2 HypothesisWorks/hypothesis#3950
imagecodecs cgohlke/imagecodecs#100
Imageio imageio/imageio#1077
JAX yes 0.4.26 google/jax#19246
Keras keras-team/keras#19691
Matplotlib yes 3.8.4 matplotlib/matplotlib#26778
MDAnalysis MDAnalysis/mdanalysis#4482
ml_dtypes yes 0.4.0 jax-ml/ml_dtypes#143
netCDF4 Unidata/netcdf4-python#1317
NetworkX yes 3.3 networkx/networkx#7390
Numba 0.60 (0.60.0rc1 is available) numba/numba#9544, Discourse post with context
numcodecs yes 0.12.1 Likely older versions too; has been stable for a while. zarr-developers/numcodecs#521
numexpr yes 2.10.0 pydata/numexpr#478
OpenCV opencv/opencv-python#943
Pandas yes 2.2.2 pandas-dev/pandas#55519
PyArrow yes 16.0 apache/arrow#39532
Pybind11 yes 2.12.0 pybind/pybind11#5009
PyData Sparse yes 0.15.2 pydata/sparse#680 (required Numba 0.60)
PyMC depends on PyTensor
PySide6 yes 6.7.0 from a comment on this issue: #26191 (comment)
PyTables PyTables/PyTables#1083
PyTensor pymc-devs/pytensor#689
Pythran yes 0.16.0 (0.15.0 works mostly, SciPy builds with it) serge-sans-paille/pythran#2189
PyTorch yes 2.3.0 pytorch/pytorch#107302
PyWavelets yes 1.6.0 PyWavelets/pywt#731
RAPIDS rapidsai/build-planning#38
Rasterio yes 1.3.10 rasterio/rasterio#3024
scikit-bio 0.6.1 scikit-bio/scikit-bio#1964
scikit-image yes 0.23.1 scikit-image/scikit-image#7282
scikit-learn yes 1.4.2 scikit-learn/scikit-learn#27075
SciPy yes 1.13.0 scipy/scipy#20375
Shapely yes 2.0.4 shapely/shapely#1972
SymPy 1.12.1
Seaborn yes 0.13.2 mwaskom/seaborn#3683
statsmodels yes 0.14.2 statsmodels/statsmodels#9194
TensorFlow (maybe in May) tensorflow/tensorflow#67291, has <2 upper bound for 2.16.1, requirements, lock file
threadpoolctl yes 3.5.0 joblib/threadpoolctl#175
tifffile yes 2024.4.24 cgohlke/tifffile#252
treelite yes 4.2.1 dmlc/treelite#560
unyt yes 3.0.2 yt-project/unyt#493
Xarray pydata/xarray#8844
XGBoost 2.1.0 dmlc/xgboost#10221
yt yes 4.3.1 yt-project/yt#4859
Zarr yes 2.18.0 zarr-developers/zarr-python#1818
zfpy LLNL/zfp#210
@jakevdp
Copy link
Contributor

jakevdp commented Apr 1, 2024

Thanks - for JAX we are planning an 0.4.26 release in the next day or two which will be built against NumPy 2.0.0rc1.

@jakevdp
Copy link
Contributor

jakevdp commented Apr 1, 2024

Also, in case you want to add it, we just released ml_dtypes v0.4.0 (https://pypi.org/project/ml-dtypes/) which is compatible with NumPy 2.0.

@jakirkham
Copy link
Contributor

Thanks for putting this together Ralf! 🙏

This is incredibly helpful 🙂

@rgommers
Copy link
Member Author

rgommers commented Apr 1, 2024

Thanks @jakevdp, I added the info for both JAX and ml_dtypes.

@neutrinoceros
Copy link
Contributor

unyt 3.0.2 was also released a couple days ago for compat with numpy 2, in case you'd like to include it !

@neutrinoceros
Copy link
Contributor

Also, here's where to track progress for cartopy : SciTools/cartopy#2339

@rgommers
Copy link
Member Author

rgommers commented Apr 2, 2024

Thanks @neutrinoceros, much appreciated. Added both packages.

@ianthomas23
Copy link
Contributor

ContourPy 1.2.1 has just been released on PyPI (https://pypi.org/project/contourpy/1.2.1/) with NumPy 2 compatibility, most relevant PR is contourpy/contourpy#371. It is a compulsory dependency of Matplotlib.

@jorisvandenbossche
Copy link
Contributor

For PyArrow, it will be the upcoming 16.0 release that will be the first numpy-2.0-compatible release, but this will only be expected in around 3 weeks at the earliest (but normally certainly before the end of the month).
For people that need a PyArrow installed in an environment with numpy 2.0, it's worth noting that there are nightly wheels available though that already work with 2.0.

@dkbarn
Copy link

dkbarn commented Apr 3, 2024

Could we add OpenCV to this list? I filed a ticket here to track progress on a numpy 2.0 build for it.

@rgommers
Copy link
Member Author

rgommers commented Apr 3, 2024

Thanks @ianthomas23, @jorisvandenbossche, @dkbarn - all info added to the table.

@hawkinsp
Copy link
Contributor

hawkinsp commented Apr 3, 2024

JAX released 0.4.26 on pypi, which is compatible with NumPy 2.0.

(NumPy folks: congratulations on your imminent v2 release!)

@ksunden
Copy link
Contributor

ksunden commented Apr 4, 2024

mpl 3.8.4 is out, built with np 2

@neutrinoceros
Copy link
Contributor

neutrinoceros commented Apr 6, 2024

yt 4.3.1 is on PyPI and built with numpy 2.0.0rc1 !

EDIT(seberg): Added to table.

@neutrinoceros
Copy link
Contributor

@h-vetinari
Copy link
Contributor

Yes, Matti pointed out that patch to me (it wasn't linked in the issue so I didn't see it), and it works fine when backported to 1.84 and even 1.82.

@jakirkham
Copy link
Contributor

jakirkham commented May 8, 2024

@rgommers could we please update these entries in the table in the OP (replacing the old ones)?

Package name Compatible release on PyPI? Min compatible version Notes
numcodecs  yes 0.12.1 Likely older versions too; has been stable for a while.
zarr-developers/numcodecs#521
Zarr yes  2.18.0  zarr-developers/zarr-python#1818

@rgommers
Copy link
Member Author

rgommers commented May 8, 2024

Updated for all recent comments, thanks all!

@jakirkham
Copy link
Contributor

jakirkham commented May 8, 2024

Thanks Ralf! 🙏

Looks like nearly all libraries have some kind of issue/PR reference (or already a working release)

One that appears to be missed is Keras, so have raised upstream issue: keras-team/keras#19691

@jakirkham
Copy link
Contributor

jakirkham commented May 9, 2024

TensorFlow is targeting a branch cut end of May, we are unsure if numpy 2.0 support will make it but we are going to try and i should have better insight next week if the update will make next release. Like Jake said TensorFlow does have a <2.0 pin in at least both 2.16.1 and 2.15.1 which are our two supported versions currently, so I don't think we should immediately break if TensorFlow does not get the update done before a 2.0 release.

Thanks @MichaelHudgins for the insight here

Am curious how things are looking for TensorFlow + NumPy 2. Were you able to get more insight in the past 2 weeks?

Edit: Have gone ahead and opened an issue ( tensorflow/tensorflow#67291 ). Let's discuss there

@h-vetinari
Copy link
Contributor

The boost version in the table is wrong. boostorg/python@0474de0 is not part of 1.85, but will be in 1.86.

PS. Sidenote: it would be helpful if the rest of the table (perhaps excluding some "most important packages") were sorted alphabetically, as it's getting a bit unwieldy to find specific packages.

@bryevdv
Copy link

bryevdv commented May 13, 2024

it would be helpful if the rest of the table (perhaps excluding some "most important packages") were sorted alphabetically, as it's getting a bit unwieldy to find specific packages.

Agreed, if "important" packages need to be specially identified, they can be bolded or a column with an X or check mark or whatever

@bersbersbers
Copy link

I could not find an issue in their bug tracker, but some testing implies that PySide6 works from version 6.7.0 on:

# exits with "A module that was compiled using NumPy 1.x cannot be run in NumPy 2.0.0rc2 as it may crash."
uv pip install numpy==2.0.0rc2 PySide6-Essentials==6.6.3.1 && python -c "from PySide6 import QtWidgets"

# works fine
uv pip install numpy==2.0.0rc2 PySide6-Essentials==6.7.0   && python -c "from PySide6 import QtWidgets"

@jakirkham
Copy link
Contributor

jakirkham commented May 16, 2024

@rgommers have a few updates batched below for the table

GDAL 3.9.0 is released (with NumPy 2 support)! 🎉

Also can we please add these NumPy 2 tracking issues to the table:

At this point Numba's 0.60.0rc1 is packaged ( numba/numba#9544 (comment) ). Working on a conda-forge RC package for it as well. Would start pointing libraries downstream of Numba to these

Boost.Python should be fixed in the 1.86.0 release (we narrowly missed 1.85.0)

Dask is looking promising ( dask/dask#11066 (comment) ). Though still want to double check things there

Lastly opened an issue for Sparse: pydata/sparse#680

@rgommers
Copy link
Member Author

Thanks all, updated for all comments.

PS. Sidenote: it would be helpful if the rest of the table (perhaps excluding some "most important packages") were sorted alphabetically, as it's getting a bit unwieldy to find specific packages.

Okay, I reordered it. The initial order has served its purpose, everything was blocked on a small set of packages at the start but those all did their release.

@hameerabbasi
Copy link
Contributor

Lastly opened an issue for Sparse: pydata/sparse#680

sparse release is up on PyPI: https://pypi.org/project/sparse/0.15.2/

@jakirkham
Copy link
Contributor

Dask 2024.5.1 works with NumPy 2

ref: https://docs.dask.org/en/stable/changelog.html#numpy-2-0-support

@rgommers
Copy link
Member Author

Great, thanks @hameerabbasi and @jakirkham. Updated the table for those, and also added RAPIDS (linking to your tracking issue @jakirkham).

We're getting there - we've had ~7 weeks since RC1 and are about 75% through the list in this issue. In the community meeting next Wednesday we should probably discuss picking a release date in June.

@bnavigator
Copy link
Contributor

What about (optional) inter-dependencies? Pandas and numexpr both claim to be compatible, but together they choke on the changed numpy 2 type repr of scalars: pandas-dev/pandas#58548

@rgommers
Copy link
Member Author

I'd see that as a regular bug - a few test failures in a combo of two packages that were not tested together in CI is common, but doesn't detract (from the purposes of this tracking issue at least) from them both being NumPy 2.0-compatible.

@rgommers
Copy link
Member Author

Posting here as well for completeness (xref #24300 (comment)): release date will be June 16th.

@peterjc
Copy link
Contributor

peterjc commented May 22, 2024

For Biopython we do have C code using NumPy, but my understanding is we are insulated from the C API changes due to using the buffer protocol rather than the NumPy C API.

I expect a "fully compatible" Biopython 1.84 in early June, but all this addresses is the scalar repr changes to work on recent Numpy v1 and the v2 release candidates biopython/biopython#4676

https://lists.open-bio.org/pipermail/biopython/2024-May/017049.html

@jakirkham
Copy link
Contributor

jakirkham commented May 22, 2024

Updated the table for those, and also added RAPIDS (linking to your tracking issue @jakirkham).

Thanks Ralf! 🙏 Also thank you for including us 🙂

@hcho3
Copy link

hcho3 commented May 28, 2024

Treelite 4.2.1 is compatible with NumPy 2.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests