DEP: deprecate scalar conversions for arrays with ndim > 0

[nschloe]

This PR reflects some of the progress achieved in issue #10404 and is used to asses the impact of the changes.

With the changes in this PR, float(numpy.array([1.0]) now gives a warning; likewise some other things:

import numpy

a = numpy.random.rand(10, 1)
a[0] = numpy.array([1.0])       # okay
a[0] = numpy.array(1.0)         # okay
a[0] = 1.0                      # okay

b = numpy.random.rand(10)
b[0] = numpy.array([1.0])       # ValueError: setting an array element with a sequence.
b[0, ...] = numpy.array([1.0])  # okay
b[0] = numpy.array(1.0)         # okay
b[0] = 1.0                      # okay

This aligns the behavior of numpy arrays with that of lists:

float([3.14]) 

TypeError: float() argument must be a string or a number, not 'list'

import numpy as np

a = np.random.rand(5)
a[0] = [3.14]

ValueError: setting an array element with a sequence.

Fixes #10404.

[eric-wieser]

Would be interesting to see how extensive the changes need to be to make CI pass, and whether this impacts the scipy or pandas test suites.

[eric-wieser]

Looks like randomgen added more tests that rely on this behavior, should be straightforward to just remove them

https://dev.azure.com/numpy/numpy/_build/results?buildId=6596&view=ms.vss-test-web.build-test-results-tab&runId=1076954&resultId=109094&paneView=debug

[nschloe]

I'm on it.

[nschloe]

So I've been working on scipy a bit (scipy/scipy#11147) and found that in parts it was quite lax when it came to float conversion of arrays. I'll fix all the inconsistencies there and in panda (if any).

All of what I discovered are actual bugs which is what makes me like this PR here even more. It will break (buggy/inefficient) code though, so there'll perhaps have to be a deprecation process.

What do you think?

[eric-wieser]

To me there's no question that if we do this, it will need to be after a deprecation period. I'm definitely in favor of tracking down bugs in downstream projects with this change, whether or not we decide to merge this. Make sure to link those fixes to here so we can get a feel for the impact.

[nschloe]

I've updated SciPy again to be warnings-free after this PR (scipy/scipy#17698). Changes are now usually minimal.

I'd be great if this PR could be looked at again. We'll soon be celebrating it's 5th birthday. :)

[asmeurer]

Are there plans to also deprecate the related reverse broadcasting behavior which was discussed on the list?

[mattip]

I marked this for triage review, it also has a conflict.

[seberg]

Planning to put this in later today and give it a shot. There is always the (correct) worry here that it might be noisy for downstream. OTOH, this has also repeatedly come up over the time.

We had brought it up very briefly yesterday, but @mattip, if you had a different intention when marking it, just let me know.

What makes me optimistic is that the scipy changes are far larger than the other libraries (all of which seem very simple, the optimizers really were most of it probably).

So to be clear: I am happy to roll it back if this annoys downstream libraries or anyone has serious concerns (libs like dask or xarray for example).
As for downstream users, I don't mind taking this deprecation slow.

[mroeschke]

@seberg in pandas, how should we treat size 1 arrays with PyObject list/tuples that we want to treat as scalars for assignment e.g.

-> values[indexer] = casted
(Pdb) values
array([10, 11, 12], dtype=object)
(Pdb) indexer
array([1])
(Pdb) casted
array([(61,)], dtype=object)
(Pdb) n
(Pdb) values
array([10, (61,), 12], dtype=object)  # want to assign a tuple
(Pdb) np.__version__
'1.24.2'

---

(Pdb) values[indexer] = casted[0]  # correct workaround for the deprecation?
(Pdb) values
array([10, 61, 12], dtype=object)  # 61 is assigned instead of (61,)

[asmeurer]

@mroeschke you can work around that by doing values[indexer] = casted[0, ...]. That forces the right-hand side to be a 0-D array instead of a scalar, which prevents NumPy from trying to call asarray on it.

[asmeurer]

Even this deprecation notwithstanding it's a good idea to use 0-D arrays instead of scalars everywhere if you're dealing with object dtype arrays with elements that themselves could be interpreted as arrays by asarray.

The issue is that casted[0] is literally type tuple (i.e., not a NumPy type anymore), so there's inherent ambiguity whether (61,) should be interpreted as the tuple object scalar or as the 1-D array array([61]) in pretty much any context where NumPy could call asarray (which is virtually everywhere).

[mroeschke]

Thanks @asmeurer. Yeah using 0D arrays in pandas seems to work for our use case

[neutrinoceros]

Provided my reading of the patch, and in particular the comment attached to the deprecation code, is correct, the error that will be raised after this deprecation is expired is TypeError ?

For context, I'm working on a case where the implicit conversion from size-1 array to scalar happens in a try: ... except TypeError: block, and landing in the except branch would be acceptable.

[seberg]

Yes, it will be a TypeError.