First value of array copy changed to zero unexpectedly

[jeff922]

CentOS 8.2.2004, Python 3.8.0, and Pycharm 2020.2
In numpy 1.19.1, the first element for bin_data_block is always zero (not expected)
In numpy 1.18.5 and earlier, the first element is non-zero and correct as it has been with prior versions.
Does not occur on Windows 10 install using same versions. Only CentOS.

import numpy

def buildElement():
    bin_data = numpy.zeros(7)

    bin_data[0] = numpy.uint64(47915728897)    #This element gets copied as zero later in code
    bin_data[1] = numpy.uint32(6e4)
    bin_data[2] = numpy.uint32(0)
    bin_data[3] = numpy.uint32(0)
    bin_data[4] = numpy.uint32(1e2)
    bin_data[5] = numpy.uint32(2e8)
    bin_data[6] = numpy.uint32(0)

    return bin_data

def main():
    bin_data_block = numpy.zeros(200, numpy.uint32)
    returnedBlock = buildElement()     #Byte 0 is correct here

    bin_data_block[0:7] = returnedBlock       #Byte 0 gets changed to zero here

    #In numpy 1.19.1, the first element for bin_data_block is always zero.  In numpy 1.18.5,
    #The first element is non-zero as it has been with prior versions.  Does not occur on
    #Windows 10 install.  Occurs in CentOS 8.2.2004, Python 3.8.0, and Pycharm 2020.2

    nop = 0

# Press the green button in the gutter to run the script.
if __name__ == '__main__':
    main()

[eric-wieser]

I think this just comes down to what (int32_t)(47915728897f) means in C - which as far as I'm aware is undefined behavior.

[seberg]

Hmmm, interesting, a simpler reproducer:

bin_data = np.array([47915728897., 0., 0.], dtype=np.float64)
print(bin_data.astype(np.uint32))  # wrap around
bin_data = np.array([47915728897., 0., 0., 0.], dtype=np.float64)
print(bin_data.astype(np.uint32))  # all 0

which switches behaviour based on length. Furthermore, this only happens for strided:

bin_data = np.array([47915728897., 0., 0., 0.] * 2, dtype=np.float64)
print(bin_data.astype(np.uint32))  # all 0
print(bin_data[::2].astype(np.uint32))  # wrap-around

All of this points to SIMD operations casting differently on the C-level.

@eric-wieser ah, was typing this, if this is undefined then the difference in vectorized and non-vectorized code is expected I guess. Now the question is whether we should try to do something about it or not.

[Qiyu8]

@seberg I got two different certain non-zero result among two systems by using your reproducer:

    bin_data = numpy.array([47915728897., 0., 0.], dtype=numpy.float64)
    print(bin_data.astype(numpy.uint32))  # wrap around
    bin_data = numpy.array([47915728897., 0., 0., 0.], dtype=numpy.float64)
    print(bin_data.astype(numpy.uint32))  # all 0

windows server 2008+X86:

[671088641         0         0]
[671088641         0         0         0]

CentOS+ARM64:

[4294967295          0          0]
[671088641         0         0         0]

Since the correct answer is:

[4294967295          0          0]
[4294967295         0         0         0]

I think both vectorized and non-vectorized code have problems.

[eric-wieser]

Since the correct answer is:

How are you determining what the correct answer is? The C standard says

When a finite value of real floating type is converted to an integer type other than _Bool, the fractional part is discarded (i.e., the value is truncated toward zero). If the value of the integral part cannot be represented by the integer type, the behavior is undefined.

Numpy perhaps doesn't do the best job of saying that it performs arithmetic mostly as if in C, but it doesn't make any guarantee that this will work either.

[seberg]

I guess the long term goal for these things might be to give a warning/error on these conversions instead, in which case the result is still undefined (and reasonably so). Similar things happen with some division related loops, where we return 0 when float would return NaN (or NaT for timedelta/datetime).

[seberg]

We now do a somewhat better job here thanks to gh-21437 many/most of these should give a warning. But when the result is well defined by an "integer overflow" occuring such a warning probably will not happen.
We do get it (on the tested platforms) for NaNs, and Infs, but not necessarily for certain value ranges.

So I think there is still an "integer overflow" problem so to speak, but not per-se a float to int cast issue anymore.

Not sure whether this should be closed, so keeping it open for now.

[seberg]

Closing, there is an other issue (too lazy to find it now) about whether this should escalate to errors (which would include the option to define it).