types: missing type information for several common functions

[henryiii]

While there have been improvements to the static types, including a few functions being added, (at least) the following functions now are completely missing types:

• np.frombuffer
• np.histogram2d
• np.histogramdd
• np.broadcast_arrays
• np.meshgrid
• np.array_split

Also these, but I think there's a reason for these:

• np.min (should use np.amin)
• np.max (should use np.amax)
• np.diag (Matlab comp function for np.diagonal)

Reproducing code example:

Use any of these function in mypy and upgrade from 1.20 to 1.21.

Error message:

A bunch of "Call to untyped function "frombuffer" in typed context" in every package I'm upgrading to use NumPy 1.21 in static typing.

NumPy/Python version information:

1.21.1, on Python 3.9.

[charris]

What mypy version are you using?

[henryiii]

0.910

[henryiii]

Here's an example: https://github.com/scikit-hep/boost-histogram/blob/53ed9ee91fdc850eb309a908e96eafc4f03442fa/.pre-commit-config.yaml#L72-L78 (UHI, Hist, and boost-histogram were all affected).

The interesting thing here is that in looking in NumPy's __init__.pyi, I see things like import diag as diag, so it seems like these were supposed to work.

For example, removing this one type: ignore in UHI (since that's a simple package to run locally, not because it's a simple situation), produces:

src/uhi/numpy_plottable.py:173: error: Call to untyped function "frombuffer" in typed context
Found 1 error in 1 file (checked 7 source files)

I see a possible problem, though. In following down frombuffer, it seems like it was imported from numpy/core/multiarray.py - but there is no .pyi file. And https://github.com/numpy/numpy/blob/v1.21.1/numpy/lib/twodim_base.pyi does not actually have any type information, just a rough template of what the parameters are. Before these were typed as "Any", so it actually hid this error, since it ignores "Any", but knows that def X is an untyped function.

So it's much less of a regression, and just a consequence of more but missing types.

[BvB93]

A bunch of "Call to untyped function "frombuffer" in typed context" in every package I'm upgrading to use NumPy 1.21 in static typing.

Per the error message: mypy is not complaining about an undefined function, it's complaining about a function whose arguments are unannotated. The relevant mypy flag, disallow-untyped-calls, is disabled by default; I'd recommend to keep it that way, as it demands a level of type coverage that partially typed libraries (such as numpy) cannot provide.

[henryiii]

Yes, I thought this was a regression, but it appears it's actually slightly better that before but still partially typed. What is the process for adding types? Is someone adding them in batches, or is it something what would help out if I were to open a PR for the functions I'm interested in? And are the types always in .pyi files (looking at the missing pyi for multiarray.py)?

[BvB93]

Yes, I thought this was a regression, but it appears it's actually slightly better that before but still partially typed. What is the process for adding types? Is someone adding them in batches, or is it something what would help out if I were to open a PR for the functions I'm interested in?

If you want an up date overview of the current typing progress I'd recommend to keep an eye on #16546 and this is also a good starting point for if anyone is interested in contributing. The hope is to wrap up introducing some (basic?) annotations for the entire library for the 1.22 release, which I think is very much feasible at the current pace.

And are the types always in .pyi files (looking at the missing pyi for multiarray.py)?

Yes, the type hints are currently exclusivvelly located in .pyi stub files (with the exception of numpy.typing itself, which mixes inline typing and stub files). As for multiarray in particular: the relevant stub file was only added back in #19060, hence why it's missing from the 1.21 release.

[BvB93]

As of #20063 all functions mentioned in this issue are have been fully annotated.