Poetry >=1.5.0 removes category from poetry.lock

[esev]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

pip (poetry)

Package manager version

poetry >=1.5.0

Language version

python 3.11

Manifest location and content before the Dependabot update

https://github.com/pywemo/pywemo/blob/main/poetry.lock

Previous format:

[[package]]
name = "cryptography"
version = "40.0.2"
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
category = "dev"
optional = false
python-versions = ">=3.6"
files = [
...

Format after Poetry 1.5.0 (lacks a category): pywemo/pywemo@65969c9 (poetry.lock)

[[package]]
name = "cryptography"
version = "40.0.2"
description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers."
optional = false
python-versions = ">=3.6"
files = [
...

dependabot.yml content

version: 2
updates:
  - package-ecosystem: "pip"
    directory: "/"
    schedule:
      # Check for updates to Poetry dependencies every week
      interval: "weekly"

  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      # Check for updates to GitHub Actions every week
      interval: "weekly"

Updated dependency

cryptography

What you expected to see, versus what you actually saw

Expected to see dependabot update for cryptography.

What happened:

Logs: https://github.com/pywemo/pywemo/security/dependabot/6/update-logs/336414827

dependabot-core/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb

Lines 300 to 304 in d097b11

	def subdep_type
	category =
	TomlRB.parse(lockfile.content).fetch("package", []).
	find { |dets| normalise(dets.fetch("name")) == dependency.name }.
	fetch("category")

updater | 2023/06/02 20:34:03 INFO Raven 3.1.2 ready to catch errors
updater | 2023/06/02 20:34:04 INFO <job_672681628> Starting job processing
updater | 2023/06/02 20:34:08 INFO <job_672681628> Starting update job for pywemo/pywemo
updater | 2023/06/02 20:34:08 INFO <job_672681628> Checking if cryptography 40.0.2 needs updating
  proxy | 2023/06/02 20:34:08 [026] GET https://pypi.org:443/simple/cryptography/
  proxy | 2023/06/02 20:34:08 [026] 200 https://pypi.org:443/simple/cryptography/
updater | 2023/06/02 20:34:09 INFO <job_672681628> Latest version is 41.0.1
updater | 2023/06/02 20:34:13 INFO <job_672681628> Sending event 3c4a16f3d63f40c7baf2887883a8854e to Sentry
  proxy | 2023/06/02 20:34:13 [028] POST https://sentry.io:443/api/1451818/store/
  proxy | 2023/06/02 20:34:13 [028] 200 https://sentry.io:443/api/1451818/store/
updater | 2023/06/02 20:34:13 ERROR <job_672681628> Error processing cryptography (KeyError)
updater | 2023/06/02 20:34:13 ERROR <job_672681628> key not found: "category"
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `fetch'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `subdep_type'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:281:in `set_target_dependency_req'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:232:in `updated_pyproject_content'
updater | 2023/06/02 20:34:13 ERROR <job_672681628> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:214:in `write_temporary_dependency_files'
...

I've also noticed dependabot adding the category to poetry.lock on PRs. category should not have been added after updating Poetry to 1.5.0.

• pywemo/pywemo@bea6994
• pywemo/pywemo@211d8f3
• pywemo/pywemo@2af241c

Native package manager behavior

The category field is not present in poetry.lock

See python-poetry/poetry#7637 where category was removed.

• pywemo/pywemo@758a71e
• pywemo/pywemo@90aee1c
• pywemo/pywemo@74a78fb

Images of the diff or a link to the PR, issue, or logs

Logs: https://github.com/pywemo/pywemo/security/dependabot/6/update-logs/336414827

Smallest manifest that reproduces the issue

No response

[phillipuniverse]

@landongrindheim I'm not so sure this is complete by simply updating Poetry, see #7418.

EDIT - yeah same problem as now caught by #7418, this issue should be re-opened.

[n-thumann]

This might still be an issue, because in greenbone/troubadix#578 Dependabot removed the category of all dependencies.
Now, when re-running Dependabot, it failed with Dependabot encountered an unknown error and the following (excerpt) showing up in the log:

[...]
updater | 2023/06/12 07:11:59 INFO <job_676691621> No update possible for pontos 23.3.5
updater | 2023/06/12 07:11:59 INFO <job_676691621> Checking if markdown-it-py 2.2.0 needs updating
  proxy | 2023/06/12 07:11:59 [232] GET https://pypi.org:443/simple/markdown-it-py/
  proxy | 2023/06/12 07:11:59 [232] 200 https://pypi.org:443/simple/markdown-it-py/
updater | 2023/06/12 07:11:59 INFO <job_676691621> Latest version is 3.0.0
updater | 2023/06/12 07:12:00 INFO <job_676691621> Sending event 1e66ef70eefd47f3866a80e81ad7cf9c to Sentry
  proxy | 2023/06/12 07:12:00 [234] POST https://sentry.io:443/api/1451818/store/
  proxy | 2023/06/12 07:12:00 [234] 200 https://sentry.io:443/api/1451818/store/
updater | 2023/06/12 07:12:00 ERROR <job_676691621> Error processing markdown-it-py (KeyError)
updater | 2023/06/12 07:12:00 ERROR <job_676691621> key not found: "category"
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `fetch'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:304:in `subdep_type'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:281:in `set_target_dependency_req'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:232:in `updated_pyproject_content'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:214:in `write_temporary_dependency_files'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:92:in `block (2 levels) in fetch_latest_resolvable_version_string'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:181:in `with_git_configured'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:91:in `block in fetch_latest_resolvable_version_string'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `block in in_a_temporary_directory'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `chdir'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/shared_helpers.rb:49:in `in_a_temporary_directory'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:90:in `fetch_latest_resolvable_version_string'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker/poetry_version_resolver.rb:62:in `latest_resolvable_version'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/python/lib/dependabot/python/update_checker.rb:42:in `latest_resolvable_version'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:74:in `preferred_resolvable_version'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:260:in `preferred_version_resolvable_with_unlock?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:252:in `numeric_version_can_update?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:202:in `version_can_update?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/common/lib/dependabot/update_checkers/base.rb:44:in `can_update?'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:204:in `requirements_to_unlock'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:88:in `check_and_create_pull_request'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:59:in `check_and_create_pr_with_error_handling'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `block in perform'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `each'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater/operations/update_all_versions.rb:34:in `perform'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/updater.rb:72:in `run'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/update_files_command.rb:38:in `perform_job'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> /home/dependabot/dependabot-updater/lib/dependabot/base_command.rb:52:in `run'
updater | 2023/06/12 07:12:00 ERROR <job_676691621> bin/update_files.rb:23:in `<main>'
updater | 2023/06/12 07:12:00 INFO <job_676691621> Checking if colorama 0.4.6 needs updating
[...]

We were also able to reproduce this behavior in another private repository and notice it, because @dependabot rebase failed.

[phillipuniverse]

More information on potential resolution added at #7418 (comment).

@esev @n-thumann - trying to figure out why this breaks in some flows but not in others. In all of my projects Dependabot distributes updates correctly. #7418 replicates the scenario in a test, but how does this work in practice? Is it that Dependabot breaks when only running with a poetry.lock file and the absence of pyproject.toml?

[esev]

I have both a poetry.lock and a pyproject.toml file. The only time I saw this error was for a Dependabot update via the Github Dependabot Vulnerability alerts under the Security tab. I continue to receive the normal PRs from Dependabot without any issues.

[phillipuniverse]

Ah! Perhaps Dependabot security checks only operate in the context of a poetry.lock. Whenever I see one of these it does always mention that the problem was in poetry.lock, not pyproject.toml.

Or maybe more likely - in the case of cryptography that usually comes in transitively, so maybe it was never managed in a pyproject.toml to begin with, and Dependabot only considers pyproject.toml if it needs to

[esev]

Or maybe more likely - in the case of cryptography that usually comes in transitively, so maybe it was never managed in a pyproject.toml to begin with, and Dependabot only considers pyproject.toml if it needs to

Interesting! I hadn't made that connection, but now that you mention it, I haven't gotten any PRs to update transitive dependencies recently

[matthias-bach-by]

That is a great observation. All of the dependencies for which Dependabot is failing me are transitive, while I keep getting updates for all my direct dependencies.

There is one important outlier, though. The PR that did remove the categories from my poetry.lock file was also a transitive one. But that was the last transitive update to succeed.

[n-thumann]

All of the dependencies for which Dependabot is failing me are transitive, while I keep getting updates for all my direct dependencies.

We noticed the same behavior: Even though the category was removed in greenbone/troubadix#578 and a we see the error from my comment above ever since, Dependabot was able to update a (direct, non-transitive) dependency in greenbone/troubadix#584 and updated poetry.lock accordingly.

Additionally, I checked the latest output and it contains:

updater | Dependabot encountered '12' error(s) during execution, please check the logs for more details.
updater | +------------------------------------+
updater | |   Dependencies failed to update    |
updater | +--------------------+---------------+
updater | | isort              | unknown_error |
updater | | astroid            | unknown_error |
updater | | semver             | unknown_error |
updater | | pylint             | unknown_error |
updater | | platformdirs       | unknown_error |
updater | | markdown-it-py     | unknown_error |
updater | | rich               | unknown_error |
updater | | setuptools         | unknown_error |
updater | | httpx              | unknown_error |
updater | | httpcore           | unknown_error |
updater | | importlib-metadata | unknown_error |
updater | | rfc3986            | unknown_error |
updater | +--------------------+---------------+

All of these 12 dependencies are transitive (checked with poetry show --tree).

[sigprof]

Running poetry lock --no-update with Poetry 1.4.2 currently works around the problem — Dependabot is able to detect the needed updates and open the corresponding PRs. But each of those PRs updates the poetry.lock file to the new format again, so the same change needs to be added to every such PR.

[phillipuniverse]

There is one important outlier, though. The PR that did remove the categories from my poetry.lock file was also a transitive one. But that was the last transitive update to succeed.

@matthias-bach-by this outlier makes sense. Dependabot checked for updates when poetry.lock was still on the pre-1.5 version, which included the category key. When Dependabot updated poetry.lock in its normal update process, Dependabot was running with Poetry 1.5 which removed the category key.

This is becoming more of a problem I got a recent CVE on grpcio and Dependabot could not generate a security update because of this transitive dependency issue (grpcio comes in for me transitively from otel).

I am fairly confident that the solution will be in #7418 but I need more input from the Dependabot maintainers to proceed.

[rachelwigell]

Hello hello! We have been having this issue too with the category lines being removed in our dependabot PRs in our repo where we use poetry. I see that v0.228.0 of dependabot-core was released yesterday with the fix from #7834 in it. I just rebased an open dependabot PR we had, and I'm able to confirm that the new poetry.lock file was generated on v0.228.0 because it says poetry 1.6.1 was used. However, I'm still seeing that the category lines are removed. Is anyone else still having issues with this today?

[deivid-rodriguez]

Hei! Unfortunately this is an existing issue, yes.

We currently don't respect the poetry version your poetry.lock uses, but instead use our own fixed version of poetry (1.6.1). This version of Poetry removes category lines from the lockfile when it finds them.

You can track resolution of this issue at #1556.

[phillipuniverse]

However, I'm still seeing that the category lines are removed. Is anyone else still having issues with this today?

@rachelwigell and just to be clear - the category key being removed is not a bug in and of itself, this is expected behavior when Poetry 1.5+ generates a lock file.

[rachelwigell]

Thanks both! I had misunderstood the issue. We can likely upgrade poetry in our repo to stop these huge diffs from being created.