deps: Lock Python indirect dependency versions

[ethanjli]

Motivation

While working on #135, I was stuck in Python dependency hell for a few days because pip chooses the versions of indirect dependencies in a way that is not reproducible ("reproducible" in the sense that if we cloned the PlanktoScope repo one day and ran the install, pip would install the same dependencies as if we cloned the PlanktoScope repo a year later and ran the install). I had problems with binary compatibility and versions of scipy & numpy (which are both direct dependencies in our project), as well as issues with a version of pandas (an indirect dependency in our project) being downloaded at a version for which pip could not find a pre-built binary from piwheels. Without having a way to lock the versions of indirect dependencies, we lack a stable foundation for making sure that our software will continue to build and run correctly in an automated or semi-automated system, even if we make no changes to the software itself.

One dirty solution could be to use pip freeze --all to make requirements.txt pin the versions of all direct and indirect dependencies, but this is undesirable because then we'd have no easy way to determine which dependencies are direct vs. indirect. A more conventional solution would be to use either Poetry or PDM for handling dependencies, instead of pip.

Goals

• Lock the version of every single Python package which needs to be installed for our Python backend to run.

Steps

• Set up poetry with a pyproject.toml file for the Python code
• Migrate the requirements.txt file into the pyproject.toml file
• Resolve any further Python dependency problems
• Reintegrate the Python scripts with the Node-RED frontend

Unresolved Questions

• Will we be able to resolve packages from PyPI for CI and/or for operation on non-RPI computers, e.g. on amd64 or aarch64? (see deps: Lock Python indirect dependency versions #138 (comment))
• Which dependencies imported from the requirements.txt file are actually indirect dependencies rather than direct dependencies?

Implementation History

• Properly manage versions of the backend's direct & indirect dependencies #175
• https://github.com/PlanktoScope/device-backend: everything from the creation of the repository up to PlanktoScope/device-backend@49241ae

[ethanjli]

Note: https://gitlab.com/openflexure/openflexure-microscope-server/-/merge_requests/124 reports issues with Poetry - they migrated to pipenv. This needs to be investigated before we can fully commit to using Poetry. Specifically, I need to try running Poetry on a Raspberry Pi to see if I encounter problems with using (or not using) piwheels, and try running Poetry in CI afterwards to ensure that it still works even after I make piwheels work on the Raspberry Pi. The problem is probably related to the poetry.lock file and file hashing, and it seems like it might've been fixed in 2022 (i.e. after the OpenFlexure project switched from Poetry to Pipenv). For more information, refer to:

• https://stackoverflow.com/questions/70564200/how-to-use-python-poetry-across-architectures
• Poetry doesn't check extra sources for package python-poetry/poetry#4854
• Multi-architecture wheel selection across disperate sources (with markers) python-poetry/poetry#5205
• Install wheel based on platform python-poetry/poetry#1616

According to https://www.piwheels.org/ and after reading https://github.com/orgs/python-poetry/discussions/7057, it looks like I need to lock on Python 3.9.