Switch packaging to PDM/Hatch/Poetry

[jakkdl]

There's some discussion in #2681, and some more in Gitter, but let's make a dedicated issue for this.

Alternatives:

1. Status quo
   • dependabot is a hassle
   • sort of impossible to properly define different package version requirements on different versions
   • regenerating *-requirements.txt locally, as is currently required to pass CI whenever changing something in *-requirements.in, requires having a matching python version and possibly other requirements on your environment.
2. Poetry
   • Does not comply with PEP 621: Support for PEP 621 python-poetry/roadmap#3
   • there's an open PR for switching to Poetry: Rework packaging to use Poetry instead. #2595
   • Supports lockfiles, quoting @njsmith on gitter:

     lockfiles for a library are very valuable for keeping CI functioning
     you don't ship them, but if you don't have them in CI and don't have very rapid development, then every time you do try to merge a PR you discover that actually CI has been broken for a while and no-one noticed so now you have to untangle that before you can finish your original thought

3. Hatch

   • Does not support lockfiles, but there are different ways to get around the above problem
     quoting @apollo13

     Fair enough. But with a nightly run you would also notice if a dependency update is breaking stuff and you could react faster by limiting the dependency till you fixed it

     quoting @agronholm

     what I do in my projects instead is cache the dependencies
     so they only get updated if I update my pyproject.toml or manually delete the caches
     => no constant dependabot churn

4. PDM
   • supports lockfiles

[apollo13]

FWIW, one should differentiate between the management tool and the build backend. For instance it is perfectly feasible to use PDM to manage the lockfile etc and hatchling (the build backend from hatch) to actually build the project. This is enabled via the following:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

and works for any project using PEP 621. In that sense it is also possible to switch between build backends easily (ie use setuptools instead of hatchling). In reality though one usually ends up with one build backend since one often uses one or two plugins to read the version from SCM etc… I like hatch/hatchling for being standards compliant and having reproducible builds by default. I am not using hatch currently though but PDM & hatchling since I often need lockfiles as well.

[Fuyukai]

I vote for Poetry purely because a) I know it b) there's no actual benefit to being standards compliant other than pointing at your pyproject and going "look, standards compliant!".

[apollo13]

I disagree on b). Standards compliance makes it easier to integrate with 3rd party tooling without having to write extra code to support what tool XYZ is doing and makes it easier to switch to another tool in the future. Knowing PDM and poetry, in my experience (and this was one of the reasons I switched) PDM is faster to address issues -- this is especially important when the dependency resolver breaks due to broken metadata or whatnot in packages (or some edge-cases with extras handling etc).

That said I don't have any real problem with either approach. If the argument ends up to choose poetry because a PR request exists for it I can offer to write a "competing" one for PDM. This way we could compare it to #2595 and see what ends up nicer -- in the end I would see moving the source code into src as an independent commit to be able to just compare the build changes. For all it is worth, I think we could switch to using src as of now with setuptools as well if wanted.

[Hnasar]

I've had great experiences using PDM. And when I've needed the extra packaging flexibility that Hatch offers, I've also happily used PDM and Hatch together -- Hatch for the build-backend, and PDM for managing the dependencies and lock file. I described this workflow here

[A5rocks]

FYI don't forget that we also have an open PR for hatchling. If a little out of date...

And pip-tools supports (standards-compliant) pyproject.toml.

[apollo13]

Well, in that sense hatchling + pip-tools is certainly the least invasive change. To that end it is probably a question how the workflows around should look like. Ie is there some need to be able to run helper scripts or similar…

// I have answered the same via email but github was down a few minutes ago, so dunno, maybe this message appears a second time.

[jakkdl]

FYI don't forget that we also have an open PR for hatchling. If a little out of date...

And pip-tools supports (standards-compliant) pyproject.toml.

Ah, found it #2449

[jakkdl]

FWIW, one should differentiate between the management tool and the build backend. For instance it is perfectly feasible to use PDM to manage the lockfile etc and hatchling (the build backend from hatch) to actually build the project.

Thanks, I hadn't appreciated the distinction... though trying to read up on it I'm still somewhat confused. I do think there's something to gain from going with a somewhat common setup, be that PDM+hatchling or something else, to make it easier for [short-term] contributors to understand and/or improve/fix bugs in the setup.

Well, in that sense hatchling + pip-tools is certainly the least invasive change. To that end it is probably a question how the workflows around should look like. Ie is there some need to be able to run helper scripts or similar…

what's the effects in practice of doing hatchling + pip-tools? E.g. does that get rid of the problems in #2681 (comment)

I'm personally starting to lean in the direction of PDM, as that seems powerful enough to support all our needs without having to resort to fancy stuff, and is widespread and well-documented.

[A5rocks]

To try to illustrate the difference between a management tool and a build backend, let me focus on a specific kind of "management tool": build frontends.

Here's a few build frontends:

• pip install .
• pip install build; python -m build
• pip install poetry; poetry build (this probably isn't technically a build frontend but... you get the vibe)
• pip install hatch; hatch build
• pip install pdm; pdm build
• python setup.py build bdist_wheel (hopefully didn't misremember that)

All of these will build a wheel, with maybe a bit more going on behind the scenes but...

Now, what's a build backend? This can be:

• pdm (maybe?)
• hatchling (not hatch)
• setuptools
• flit

Both these are obviously non-exhaustive, but the big difference is a build backend is something specific by PEP ??? (forgot the number :( ) and gets called by the build frontend through a specified API (this is why python -m build works) to make the wheel.

---

In my personal opinion, I don't like using a "frontend" that I have to install globally or use consistently: I don't want to deal with that and it adds some level of ambiguity IMO. Don't let that stop us from using one if it's the best choice, just wanted to offer my opinion :P

[apollo13]

though trying to read up on it I'm still somewhat confused. I do think there's something to gain from going with a somewhat common setup, be that PDM+hatchling or something else, to make it easier for [short-term] contributors to understand and/or improve/fix bugs in the setup.

Oh agreed, but I guess it is important to define what one wants to achieve. For instance PDM (and I think poetry as well) support custom scripts so you can say pdm run tests to execute the tests etc… Hatch goes even further allowing you to define environments and scripts for tests/docs allowing one to get rid of tox, makefiles and whatnot. So in the end the management tooling simplifies tasks while the build backend solely has the task of building an sdist/wheel. Given that most management tools shipped their own build backends (at least before most pyproject keys got standardized) it is easy to get confused by the tool vs backend. No matter which management tool or build backend one uses python -m build or pip install . will roughly create a new venv and install the build backend in it and then execute that to actually build the package.

what's the effects in practice of doing hatchling + pip-tools? E.g. does that get rid of the problems in #2681 (comment)

As far as I can tell no, the moment you generate a requirements.txt from all those tools you are limiting yourself to one version of python / operating system and generally loose all the "optional" stuff that is required for windows or older python versions (I am not a user of pip-tools so I might be wrong there…).

Now, what's a build backend? This can be:

Flit is not build backend, flit_core is. setuptools is a bit in the middle, it's build backend is setuptools.build_meta. PDM has a namespaced (I think) package pdm.backend shipped via pdm-backend

In my personal opinion, I don't like using a "frontend" that I have to install globally or use consistently

Oh yeah, there are ups and downs to everything. The good thing is noone forces you to use a specialized "frontend" as long as you are fine using pip install . or python -m build -- and I guess you'll have at least pip installed :) That said for anything aside from an installation you will need to use the frontend the project chooses since lockfiles etc are not standardized yet. Then again (and this is just my personal opinion) I find it easier for newcomers to say something along the lines of: "run pdm test" and it will do the correct thing as opposed to the current docs which say:

cd path/to/trio/checkout/
pip install -r test-requirements.txt  # possibly using a virtualenv
pytest trio

This alone already provides multiple options like running in a venv or not which leads to more potential issues.

What is unique here for hatch (which sadly doesn't have lockfile support yet) as far as I know is that it acknowledges that different tasks (building docs or running tests) requires different environments and dependencies. While you can have pdm/poetry run test you'd generally install the dependencies as part of the dev dependency list into the same venv where your actual code lives. In rare situations this can cause problems because the sphinx you need to build your docs conflicts with the deps from your app (but that doesn't happen often)

[gdiscry]

A recent entry in this space is Pyprojectx (disclosure: never used it, just heard about it).

It looks like it fills the need that some dev tools need their own environment with a separate dependency tree, while remaining reproducible.

Therefore you can have:

1. build backend: hatchling, pdm-backend, flit-core, setuptools
2. dev dependency manager (for the tests): pdm, poetry (with lockfile)
3. dev tools manager (sphinx, black, mypy, …): pyprojectx

[apollo13]

There is also a new article about some experiences with Hatch: https://andrich.me/2023/08/switching-to-hatch/

[jakkdl]

A recent entry in this space is Pyprojectx (disclosure: never used it, just heard about it).

So kind of like tox?

That said I don't have any real problem with either approach. If the argument ends up to choose poetry because a PR request exists for it I can offer to write a "competing" one for PDM. This way we could compare it to #2595 and see what ends up nicer

We do have about a million PR's open messing with pretty much every single part of this project, but I think you can go ahead with this and we can start tinkering with it to see how it works out and how much of the test scripts/CI/etc it affects.

[altendky]

Does pip-tools now provide cross-platform aware locking? I haven't been tracking the options lately, so I haven't looked at PDM, but that was kind of the sealing point for me with Poetry. If I use Poetry I can drop my custom (and silly) cloud-run pip-tools locking for each platform. jazzband/pip-tools#826 might be the relevant issue at this point.

[jakkdl]

not afaik, I think I found that same issue researching #2681 (comment)

[jakkdl]

Re @A5rocks in #2835

Just popping by to mention that #2790 probably will end up not making a change to how we have requirements files [...]
But I still don't know if we can have platform-independent lockfiles using pip-tools, which is my biggest gripe with it.

My biggest gripes are the requirements files in particular, from the OP:

• sort of impossible to properly define different package version requirements on different versions
• regenerating *-requirements.txt locally, as is currently required to pass CI whenever changing something in *-requirements.in, requires having a matching python version and possibly other requirements on your environment.

so if switching tooling doesn't get rid of *-requirements.txt I'm much much less excited by it.

(at least if I have my way and there isn't a great argument for it -- I really don't like having to install a 3rd party tool globally to install requirements ... or have to make 2 venvs, one for the tool and one for the project + deps).

I feel like this shouldn't be necessary - but possibly depends on the tool we switch to. If it's a tool that also manages virtualenvs it might be needed globally, but otherwise you should be able to 1. create a virtualenv, 2. install the packaging tool in it, 3. install the package. And IIRC you shouldn't need to do step 2 explicitly either.

[richardsheridan]

• regenerating *-requirements.txt locally, as is currently required to pass CI whenever changing something in *-requirements.in, requires having a matching python version and possibly other requirements on your environment.

What I was imagining in the other issue is that we could just ping pre-commit's bot to regenerate *requirements.txt whenever necessary, so it never has to be touched locally. It would not directly solve the different python versions thing so maybe it would only be a stopgap solution at most.

[A5rocks]

I think getting rid of setup.py is in itself a worthwhile goal!

But I agree with the python version thing -- that's my main problem with pip-compile specifically ATM. I don't think that problem necessarily needs to exist, just it's an implementation issue. Maybe I'm wrong.

[webknjaz]

Does pip-tools now provide cross-platform aware locking? I haven't been tracking the options lately, so I haven't looked at PDM, but that was kind of the sealing point for me with Poetry. If I use Poetry I can drop my custom (and silly) cloud-run pip-tools locking for each platform. jazzband/pip-tools#826 might be the relevant issue at this point.

Yes, it's hard. Though I now prefer what I described in a comment there — jazzband/pip-tools#826 (comment) — a matrix of lockfiles. Sure, it's not ideal from the number of files perspective, but it's quite straightforward to understand otherwise. And with tooling (like I have for tox there) that selects proper lockfiles for the env, it's not that bad, and the CI stability is just fantastic!

Generating deptrees is hard, especially for things that cannot physically be built on the platform where you run the resolver. I think, it's possible if the entire tree has wheels published for all the platform in the matrix, but as soon as there's no matching wheel and the resolver is forced to build from sdist, that starts influencing the output of the depresolver dynamically. There's no way around it, regardless of what env management tool you choose. And I've found that having a matrix of pip-native constraints is easier, than it seems.

In other news, I've just merged a PR in pip-tools the other day, that allows pinning the build deps. Normally, people concentrate on the runtime dependencies, overlooking that there are things that aren't pinned during the build. Hopefully, that'll be addressed soon.

I'm sure, this project needs tox for env management + lockfiles per each supported env, not some fancy CLIs that attempt to manage more than they should on devs' systems.

As for the backends, I've heard that hatchling is nice (for pure-Python projects, of course), but setuptools provides everything necessary, so there's no real benefit in replacing it.

[webknjaz]

@altendky I just posted an explainer about the things I think people misunderstand regarding the constraint files, in case you're curious: jazzband/pip-tools#1326 (comment).

[ofek]

Maintainer of Hatch/Hatchling here! Let me know if I can be of assistance in reviews or a PoC. FYI if you want to use more than the backend (I saw concerns about lock files) then there is actually a nice plugin that does locking https://github.com/juftin/hatch-pip-compile

[webknjaz]

That looks cool! I wonder if there's something similar for tox so I wouldn't have to write my own thing.

[ofek]

Ah I see, is this org sort of locked into tox?

edit: hmm, I don't see tox

[webknjaz]

No, but I am.

[ofek]

This was merged #2860

I would be curious in hearing feedback as to why the original approach of Hatchling was not pursued. Any feedback I get gets incorporated to make it easier for everyone! My goal is for Hatchling to be the best and easiest choice for every use case (right now extension modules is the only hang up). I now maintain a comparison page which was lacking before, perhaps documentation was a factor https://hatch.pypa.io/latest/why/#build-backend

[A5rocks]

I don't think it's anything specific against hatchling, moreso that the other pr which was just closed was in limbo.

But we're still considering hatchling, I think? Main point against it right now is that it doesn't have something better than pip-compile as a lock file...

(But we haven't chosen poetry yet at least IMO cause that dictates a lot like CLI tool and Python-requires stuff)