switch dependency management and package installation to poetry

[bhearsum]

I consider this an experiment to see if Poetry might be better than our current set-up. The main pros are:

• All dependencies are managed in pyproject.toml, with the pinned versions ending up in poetry.lock. Bumping versions is trivial - we simply run poetry lock and it will regenerate the latter from the former.
• Better dependency resolver. Poetry will refuse to continue if a package if it is incompatible with any of its peers. (Unlike pip, which will choose one, and then yell at you that it's incompatible with something else.)
• Completely eliminating dependence on pip, setuptools, and other things that seem to have so many sharp edges (largely due to being around for so long).

There are a couple of downsides:

• We have to bootstrap Poetry before we can use it. I haven't found any official images with it pre-installed.
• I'm not certain if Dependabot will still work. https://docs.github.com/en/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file claims it will, but there seems to be glaring issues such as Poetry version support dependabot/dependabot-core#1556.

I did my best to avoid changing anything else as part of this, but there was a couple of things I ended up doing:

• Reformatting with the latest black version
• Slight rework of init.sh to avoid depending on environment variables set in Dockerfiles. This might not actually be necessary in retrospect, but I had trouble with the env vars in an earlier iteration at the very least. It also paves the way to unify the admin & public Dockerfiles (as we do with Balrog).
• Dropped having different groups for admin and public. Technically we could maintain this, but there's no real downside to public having the admin-only dependencies. (It does make the image a bit smaller to do so, but it also means we have two distinct images...so maybe it's a wash?)

[ahal]

Nice, I like it!

[ahal]

I realize this is just copying what we previously had, so definitely not a consideration for this PR.. But we may want to consider using the compatible release operator here. Especially if we're going to let dependabot run loose with it. Otherwise we may end up pulling in major versions for multiple deps at a time which could make it difficult to track down bustage.

Then again, using compatible release might mean that we simply don't ever upgrade things which can also be problematic. So I'm not saying one way is wrong and the other isn't, more of some food for thought.

[bhearsum]

At a high level, I think it's good allowing dependabot (or whatever) to attempt to upgrade packages unless we know they aren't compatible -- that way at least we get some feedback on the PR or PRs that are opened, and it makes it easy to pull in simple upgrades on a regular basis.

I agree that it's worth talking about this again soon though - it's been a long time since we've had an explicit conversation about how we manage this.

[ahal]

This is fine, just noting that if you want to avoid using poetry's virtualenvs and continue using the same Python installations as before (e.g to avoid poetry run later on), you can use poetry export to convert the poetry.lock into a requirements.txt and then pip install that with whatever Python you want.

[bhearsum]

And presumably this avoids a lot of the pip dependency resolver (and other) pitfalls, since pip is really just doing download + install (no resolution, etc.)?

[gbrownmozilla]

This looks great. Any concerns about using poetry for all releng projects?

[bhearsum]

This looks great. Any concerns about using poetry for all releng projects?

I was thinking of this as a way to trial idea. I certainly have no concerns from the outset, and no objections if someone wants to similarly switch other projects. But I also think it's fine to stay the course with pip-compile-multi until we see how Poetry works in practice for Ship It for a bit.

[jcristau]

Could be a followup, but if the only difference between the admin and public Dockerfiles is the CMD then we could have just one of them, and set the command in docker-compose.yml?

[bhearsum]

Yeah, we should do that (as we do in Balrog). I was trying to contain the scope/risk creep for now 😛

[bhearsum]

This part looks like it will require coordination with a cloudops-infra change, as we don't explicitly set the entrypoint nor command for our deployments. I filed #1310 for it.