TST: Nightly wheel asking for setuptools version that Actions does not have

[pllim]

numpy/pyproject.toml

Line 5 in 368baf4

"setuptools==59.2.0",

resulted in such errors in downstream jobs that grabs nightly numpy wheel

Ignoring packaging: markers 'platform_machine == "arm64"' don't match your environment
ERROR: Could not find a version that satisfies the requirement setuptools==59.2.0 (from versions: none)
ERROR: No matching distribution found for setuptools==59.2.0

https://github.com/astropy/astropy/runs/6818500953?check_suite_focus=true (GitHub Actions ubuntu-latest, Python 3.8)

This setup has been working for a while but suddenly stopped working. I cannot tell if something changed in Actions or in the nightly wheel.

[charris]

Which arm64 is that? Apple?

[charris]

We install setuptools from pypi. Are you/ getting them some other way?

[pllim]

It is supposed to be ubuntu-latest. I don't think we explicitly install setuptools but it wasn't a problem before. Maybe let me try an explicit install and get back to you.

[pllim]

Hello again. I am unable to get the test env to pip install the exact setuptools versions that your pyproject.toml wants. Is there a reason why it must be that exact version? See astropy/astropy#13322

[charris]

@pllim, other versions work, but setuptools keeps changing things in distutils that breaks numpy distutils, which is currently needed for C, C++, and Fortran. If you cannot pip install that version, something is broken. One possibility that comes up now and then is you should always use the pip module python -mpip ... just to be sure pip puts things in the correct site-package.

[mattip]

I commented on that issue. Something is off when using the "nightly" builds at https://pypi.anaconda.org/scipy-wheels-nightly: the workflow is building from source rather than using a pre-built wheel. Then the build from source tries to find setuptools at https://pypi.anaconda.org/scipy-wheels-nightly/simple, which of course will fail.

[saimn]

Probably because wheels are missing for the latest numpy dev version, the Actions job is installing Numpy from source :

Collecting numpy
  Downloading https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/1.24.0.dev0%2B163.g5fedeb7e4/numpy-1.24.0.dev0%2B163.g5fedeb7e4.tar.gz (10.8 MB)

In https://pypi.anaconda.org/scipy-wheels-nightly/simple/numpy/ the wheels are present for dev0+102 and there is only the tarball for dev0+163.

[mattip]

Makes sense. Maybe fallout from #21706 where the wheel building failed. One theory is that the sdist build succeeded and uploaded. I will trigger a wheel upload, so in a few hours they should appear.

[mattip]

Maybe we should add explicit instructions in the downstream package authors page to use --only-binary

pip install -U --pre --only-binary :all: -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy

[mattip]

@lithomas1 is there a way to tie the sdist job to the success of the wheel job so an sdist will not be uploaded if the wheel build fails?

[charris]

I suppose there are two we should do to handle this in the future:

1. Delete the source upload.
2. Trigger a new build when the fix goes in.

[lithomas1]

@lithomas1 is there a way to tie the sdist job to the success of the wheel job so an sdist will not be uploaded if the wheel build fails?

It's possible(by making the sdist job dependent on the wheel one), but I don't think that is a good idea.

The whole point is to install from the sdist if there are no wheels available. This is useful for downstream(e.g. if we want to test on Python 3.11 for which no wheels are available, or on more exotic platforms I guess). Without the sdist, people would have to do a pip install git+... from numpy main, in those cases, which is not ideal.

In the future, I'd recommend aligning the regular testing CI(make the regular CI run the full suite) with the wheels CI to prevent this from happening.

I can also make the Wheel Builders report failure via an issue by modifying the action if that is desired too.

[mattip]

The whole point is to install from the sdist if there are no wheels available

We have two types of wheels: "nightly" ones that are uploaded to one URL, and the "official" ones that are uploaded to PyPI. I agree that the official ones should have an sdist.

Here we are discussing the "nightly" ones. The point of them is to allow downstream projects to conveniently test latest NumPy HEAD. Uploading only an sdist for nightlies causes disruption to the desired CI action since it is non-trivial to correctly build NumPy from source.