Conda support

[benjyw]

Kicking off a discussion about what it means for Pants to support Conda environments. I.e., what users would expect from such a thing, and how we can implement it.

[benjyw]

As a first approximation, in the Python domain at least, we can think of a conda environment as an interpreter with some extra stuff in its site-packages, I think? So maybe all that is needed is a way to ensure that this interpreter is used?

[henridwyer]

If we were able to specify a conda environment to use for a given resolves, this would fairly easily allow using conda and pants together, even if the conda environments themselves would need to be managed outside of pants (as a first pass support)

[benjyw]

How would we specify the conda environment in a machine-independent way?

[benjyw]

We can't use a local path, for example, because that would vary across machines.

[thejcannon]

conda also has system libraries. Dunno if those are added to site-packages though

[benjyw]

Right, but we're assuming that the conda interpreter already knows how to use those

[bmcandr]

I use conda with Pants currently. As long as you conda activate <env-name before executing any Pants goals, Pants will find the Python interpreter in the conda environment. Where we run into trouble is third-party dependency inference. Pants doesn't recognize conda env files so we can't pass them to python_requirements like we can with
a requirements.txt. As a result we have to add individual python_requirements for third-party dependencies, effectively duplicating our env file which is not ideal. Therefore, support for conda env files as sources for python_requirements would be very welcome.

[benjyw]

I see, so a conda_env_requirements() target generator that generates python_requirement targets out of an env file? That seems fairly straightforward if anyone wants to tackle it.

[bmcandr]

Possibly. Things can get a little tricky in conda land as it isn't purely a Python package manager. This comment explains it better than I can. For example, our conda env contains a mix of dependencies installed from conda sources and pip sources. But, Pants is awesome and can probably handle this in some way.

I'm a Pants noob and fairly busy at the moment, but if I were to look into adding a target generator for conda where would I start? Can you point me to something similar in the Pants repo that I can study?

[benjyw]

This is where we turn a python_requirements() target wrapping a requirements.txt into a bunch of individual python_requirement targets: https://github.com/pantsbuild/pants/blob/main/src/python/pants/backend/python/macros/python_requirements.py

And that dir also has poetry_requirements.py and pipenv_requirements.py which are similar for pyproject.toml and Pipenv.lock respectively. So there should be some helpful pattern matching on those three...

[salotz]

I would like support for doing conda builds integrated to Pants, at least for pure-python. I had this completely automated in the past just to pull from sdists and autogenerate the conda recipes (not using Pants).

I'll be doing something similar hopefully with Pants soon and can report my experience.