📚 todo: Provide compiled Cython/C/C++ binaries

[dcmvdbekerom]

💭 Description

Over the past years RADIS has been heavily optimized. Some of these optimization rely on cython/c++ code which is compiled during installation of RADIS. However, we shouldn't expect all users to have a compiler installed, and we also don't want them to miss out on optimizations. The situation is even worse for GPU code, as it currently requires cupy, which requires the CUDA Toolkit (3GB), which requires Microsoft Visual Studio (10+GB).

To prevent users have to compile code, we should do it for them. The pre-built libraries should be "staged", because we also don't want developers to have to rebuild all code if they're only working on one thing. For example, if a dev works on a cython optimization, we shouldn't require them to also be able to compile GPU code (requiring CUDA Toolkit and MSVC).

Cython/C++ code could be compiled into pre-built wheels. The problem with Wheels is that we need a separate version for OS (windows/linux/macOS), architecture (32/64bit), and Python version. However, there is a package called cibuildwheels that let's the CI build all these different flavors. I propose we build these whenever a version is released to PyPy.

Currently all cython functions have a python backup in case compilation fails. It would be better if we provide the backup in the form of Wheels, because then everyone will be running the same code. The disadvantage is that the barrier for development is slightly higher, because it would require more code to be re-compiled during development.

Curious to hear thoughts & suggestions!

Edit: CUDA can be handled entirely without C/C++ compilation, so I made a separate thread here: #603)

[erwanp]

I'm curious about how it is done and implemented in other packages. Do you know of any gpu-accelerated Python package that doesn't require the C++ compiler & CUDA Toolkit ? Their devs could help us make the right choices in terms of deployment

[dcmvdbekerom]

It seems at least for the C++ part, cibuildwheels is how some of the larger packages do it
For the GPU part, most packages I know of require the CUDA toolkit (e.g., Vaex is built on top of cupy, which requires the CUDA toolkit)

E: Looks like PyTorch has CUDA pre-compiled

[erwanp]

Btw the deploy step of Radis (activated on pushes to the master branch) already builds a wheel https://github.com/radis/radis/blob/develop/.travis.yml#L84

[dcmvdbekerom]

Btw the deploy step of Radis (activated on pushes to the master branch) already builds a wheel

That's interesting.. It seems it makes one wheel for all platforms and versions.
Unfortunately the build cython files (.pyd) are speficic to each python version and platform. This is where cibuildwheels is useful because it builds for all (selected) versions and platforms.

I'm thinking we could use cibuildwheels to compile cython code whenever it is modified during development, and then pack the .pyd files with the bdist_wheel built by travis.

[erwanp]

Said today : we'll release next version 0.15 with cibuildwheels and see if it works ! I'm taking this.

[dcmvdbekerom]

We will have to decide: what is the lowest python version we would want to support?