ENH: Symbolic solver for dimension specifications.

[pearu]

Resolves #8062

Possibly conflicting with #19388 that replaces shape with f2py_shape. Whichever PR lands last, should update the usage of shape in crackfortran.py.

[mattip]

LGTM give 6 new alerts for this PR. Are they accurate?

This pull request introduces 6 alerts when merging 26752c3 into 0656fc4 - view on LGTM.com
new alerts:

3 for Variable defined multiple times
1 for Comparison using is when operands support __eq__
1 for Unused local variable
1 for Incomplete ordering

[pearu]

LGTM give 6 new alerts for this PR. Are they accurate?

The LGTM alerts were partially accurate but were certainly useful to pinpoint one real bug. Fixed the bug and warnings in the latest commit.

[mattip]

+1 for LGTM

[HaoZeke]

This looks fantastic! I will recheck the issues to see if this fixes more than the one referenced here; but this is a huge step forward.

[pearu]

scipy builds and all its tests pass with this PR.

[charris]

@pearu Could you add a release note for this in doc/release/upcoming_changes? This is probably a new_feature or improvement.

[melissawm]

Very tiny typos otherwise this looks great, thanks @pearu !

[melissawm]

Suggested change

	"""This test site tests various expressions that are used as dimension
	"""This test suite tests various expressions that are used as dimension

[melissawm]

Suggested change

	that a developer specifies the corresponing C expressions in a
	that a developer specifies the corresponding C expressions in a

[charris]

Thanks @pearu. I've done the spelling fixes and added a release note in #19961, feel free to edit the release note. Does #19388 need fixes?

[risicle]

Hmm. Ever since commit 88f988a introduced here, in NixOS, the scikit-learn 1.0.2 kernel PCA unit tests cause crashes for us which appear to be related to a corrupted heap. This is with joblib 1.1.0, scipy 1.8.0, threadpoolctl 3.0.0, gfortran 9.3.0, gcc 10.3. Still investigating.

Edit: also tried with numpy 1.22.2 including fix #20724

[melissawm]

@pearu thoughts?

[pearu]

@risicle Can you provide a minimal reproducer of the reported crashes? The reproducer should include the f2py wrapping step as the messages there may give a hint of what goes wrong.

[risicle]

I'll try - it's all inside scipy's flapack_dsyevr, so I'll have to kick scipy's build process a bit.

Currently trying to convince our lapack to build with debugging symbols..

[risicle]

A lot of fighting with debug symbols and a crash course in fortran later, it looks like what is happening is dsyevr is being called with an M of 10, but a ISUPPZ array allocated only one element long (4 bytes).

I'll try to get scipy's f2py calls to give up their secrets next.

[risicle]

https://gist.github.com/risicle/d7450ec92bd89bff138ead438c28ce1c
https://gist.github.com/risicle/0e201ea432ffda2d0015a51daec88e7f

@@ -49783,33 +49783,33 @@
   capi_iwork_tmp = array_from_pyobj(NPY_INT,iwork_Dims,iwork_Rank,capi_iwork_intent,Py_None);
   if (capi_iwork_tmp == NULL) {
     PyObject *exc, *val, *tb;
     PyErr_Fetch(&exc, &val, &tb);
     PyErr_SetString(exc ? exc : _flapack_error,"failed in converting hidden `iwork' of _flapack.dsyevr to C/Fortran array" );
     npy_PyErr_ChainExceptionsCause(exc, val, tb);
   } else {
     iwork = (int *)(PyArray_DATA(capi_iwork_tmp));

   /* Processing variable z */
-  z_Dims[0]=(compute_v?MAX(0,n):0),z_Dims[1]=(compute_v?(*range=='I'?iu-il+1:MAX(1,n)):0);
+  z_Dims[0]=(compute_v ? MAX(0, n) : 0),z_Dims[1]=(compute_v ? (*range == 'I' ? 1 - il + iu : MAX(1, n)) : 0);
   capi_z_intent |= F2PY_INTENT_OUT|F2PY_INTENT_HIDE;
   capi_z_tmp = array_from_pyobj(NPY_DOUBLE,z_Dims,z_Rank,capi_z_intent,Py_None);
   if (capi_z_tmp == NULL) {
     PyObject *exc, *val, *tb;
     PyErr_Fetch(&exc, &val, &tb);
     PyErr_SetString(exc ? exc : _flapack_error,"failed in converting hidden `z' of _flapack.dsyevr to C/Fortran array" );
     npy_PyErr_ChainExceptionsCause(exc, val, tb);
   } else {
     z = (double *)(PyArray_DATA(capi_z_tmp));

   /* Processing variable isuppz */
-  isuppz_Dims[0]=(compute_v?(2*(*range=='A'||(*range=='I' && iu-il+1==n)?n:0)):0);
+  isuppz_Dims[0]=(compute_v ? 2 * (*range == 'A'||(*range == 1 + 'I' && iu - il == n) ? n : 0) : 0);
   capi_isuppz_intent |= F2PY_INTENT_OUT|F2PY_INTENT_HIDE;
   capi_isuppz_tmp = array_from_pyobj(NPY_INT,isuppz_Dims,isuppz_Rank,capi_isuppz_intent,Py_None);
   if (capi_isuppz_tmp == NULL) {
     PyObject *exc, *val, *tb;
     PyErr_Fetch(&exc, &val, &tb);
     PyErr_SetString(exc ? exc : _flapack_error,"failed in converting hidden `isuppz' of _flapack.dsyevr to C/Fortran array" );
     npy_PyErr_ChainExceptionsCause(exc, val, tb);
   } else {
     isuppz = (int *)(PyArray_DATA(capi_isuppz_tmp));