poetry: failure to widen range (^11.0.3 => ^12.0)

[msw-kialo]

What would you like help with?

I think I found a bug

How are you running Renovate?

Mend Renovate hosted app on github.com

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

No response

Please tell us more about your question or problem

Given the upgrade 11.0.3 to 12.0, Renovate fails to update such poetry dependency if they are locked with range: apparently, Renovate thinks ^11.0.3 would include 12.0 and leaves the pyproject.toml unchanged.

Reproduction repo: https://github.com/msw-kialo/renovate-poetry-websockets (however, it is not much to see as Renovate won't open a PR as it is empty).

Logs (if relevant)

Logs

DEBUG: packageFiles with updates
{
  "baseBranch": "master"
  "config": {
    "poetry": [
      {
        "deps": [
          {
            "commitMessageTopic": "Python",
            "currentValue": "~3.12",
            "currentVersion": "3.12.3",
            "datasource": "github-releases",
            "depName": "python",
            "depType": "dependencies",
            "managerData": {
              "nestedVersion": false
            },
            "packageName": "containerbase/python-prebuild",
            "registryUrl": "https://github.com",
            "registryUrls": null,
            "sourceUrl": "https://github.com/containerbase/python-prebuild",
            "versioning": "poetry",
            "warnings": [],
            "updates": []
          },
          {
            "changelogUrl": "https://websockets.readthedocs.io/en/stable/project/changelog.html",
            "currentValue": "^11.0.3",
            "currentVersion": "11.0.3",
            "datasource": "pypi",
            "depName": "websockets",
            "depType": "dependencies",
            "fixedVersion": "11.0.3",
            "isSingleVersion": true,
            "lockedVersion": "11.0.3",
            "managerData": {
              "nestedVersion": false
            },
            "packageName": "websockets",
            "registryUrl": "https://pypi.org/pypi",
            "sourceUrl": "https://github.com/python-websockets/websockets",
            "versioning": "poetry",
            "warnings": [],
            "updates": [
              {
                "bucket": "major",
                "newVersion": "12.0",
                "newValue": "^11.0.3",
                "releaseTimestamp": "2023-10-21T14:19:28.000Z",
                "newMajor": 12,
                "newMinor": 0,
                "updateType": "major",
                "isRange": true,
                "isLockfileUpdate": true,
                "branchName": "renovate/websockets-12.x-lockfile"
              }
            ]
          }
        ],
        "extractedConstraints": {
          "python": "~3.12"
        },
        "lockFiles": [
          "poetry.lock"
        ],
        "packageFile": "pyproject.toml",
        "packageFileVersion": "0.1.0"
      }
    ]
  }
}

DEBUG: syncBranchState() (branch="renovate/websockets-12.x-lockfile")
DEBUG: getBranchPr(renovate/websockets-12.x-lockfile) (branch="renovate/websockets-12.x-lockfile")
DEBUG: findPr(renovate/websockets-12.x-lockfile, undefined, open) (branch="renovate/websockets-12.x-lockfile")
DEBUG: findPr(renovate/websockets-12.x-lockfile, undefined, closed) (branch="renovate/websockets-12.x-lockfile")
DEBUG: branchExists=false (branch="renovate/websockets-12.x-lockfile")
DEBUG: dependencyDashboardCheck=other (branch="renovate/websockets-12.x-lockfile")
DEBUG: Check for closed PR because recreating closed PRs is disabled. (branch="renovate/websockets-12.x-lockfile")
DEBUG: findPr(renovate/websockets-12.x-lockfile, Update dependency websockets to v12, !open) (branch="renovate/websockets-12.x-lockfile")
DEBUG: prAlreadyExisted=false (branch="renovate/websockets-12.x-lockfile")
DEBUG: Checking schedule(at any time, null) (branch="renovate/websockets-12.x-lockfile")
DEBUG: No schedule defined (branch="renovate/websockets-12.x-lockfile")
DEBUG: Branch needs creating (branch="renovate/websockets-12.x-lockfile")
DEBUG: Using reuseExistingBranch: false (branch="renovate/websockets-12.x-lockfile")
DEBUG: Setting current branch to master (branch="renovate/websockets-12.x-lockfile")
DEBUG: Initializing git repository into /tmp/renovate/repos/github/msw-kialo/renovate-poetry-websockets (branch="renovate/websockets-12.x-lockfile")
DEBUG: Performing blobless clone (branch="renovate/websockets-12.x-lockfile")
DEBUG: git clone completed (branch="renovate/websockets-12.x-lockfile")
{
  "durationMs": 686
}

DEBUG: latest repository commit (branch="renovate/websockets-12.x-lockfile")
{
  "latestCommit": {
    "hash": "7a02e022b84a2cf1c2ce359558d0659700311922",
    "date": "2024-04-23T15:51:35+02:00",
    "message": "[tooling] Test websocket upgrade",
    "refs": "HEAD -> master, origin/master, origin/HEAD",
    "body": "",
    "author_name": "Malte Swart",
    "author_email": "msw@kialo.com"
  }
}

DEBUG: latest commit (branch="renovate/websockets-12.x-lockfile")
{
  "branchName": "master"
  "latestCommitDate": "2024-04-23T15:51:35+02:00"
}

DEBUG: manager.getUpdatedPackageFiles() reuseExistingBranch=false (branch="renovate/websockets-12.x-lockfile")
DEBUG: poetry.updateLockedDependency: websockets@11.0.3 -> 12.0 [poetry.lock] (branch="renovate/websockets-12.x-lockfile")
DEBUG: updateArtifacts for nonUpdatedPackageFiles (branch="renovate/websockets-12.x-lockfile")
DEBUG: poetry.updateArtifacts(pyproject.toml) (branch="renovate/websockets-12.x-lockfile")
DEBUG: Updating poetry.lock (branch="renovate/websockets-12.x-lockfile")
DEBUG: Using python version from pyproject.toml (branch="renovate/websockets-12.x-lockfile")
DEBUG: Using poetry version from poetry.lock header (branch="renovate/websockets-12.x-lockfile")
DEBUG: Setting CONTAINERBASE_CACHE_DIR to /tmp/renovate/cache/containerbase (branch="renovate/websockets-12.x-lockfile")
DEBUG: Using containerbase dynamic installs (branch="renovate/websockets-12.x-lockfile")
DEBUG: Resolved stable matching version (branch="renovate/websockets-12.x-lockfile")
{
  "toolName": "python"
  "constraint": "~3.12"
  "resolvedVersion": "3.12.3"
}

DEBUG: Executing command (branch="renovate/websockets-12.x-lockfile")
{
  "command": "install-tool python 3.12.3"
}

DEBUG: exec completed (branch="renovate/websockets-12.x-lockfile")
{
  "durationMs": 17757
  "stdout": "installing v2 tool python v3.12.3\n[14:04:51.310] INFO (51): Installing tool python@3.12.3...\nlinking tool python v3.12.3\nPython 3.12.3\npip 24.0 from /opt/containerbase/tools/python/3.12.3/lib/python3.12/site-packages/pip (python 3.12)\n[14:05:08.198] INFO (51): Installed tool python in 16.8s.\n"
  "stderr": ""
}

DEBUG: Executing command (branch="renovate/websockets-12.x-lockfile")
{
  "command": "install-tool poetry 1.8.2"
}

DEBUG: exec completed (branch="renovate/websockets-12.x-lockfile")
{
  "durationMs": 25847
  "stdout": "[14:05:09.179] INFO (213): Installing tool poetry@1.8.2...\ninstalling v2 tool poetry v1.8.2\nlinking tool poetry v1.8.2\nPoetry (version 1.8.2)\n[14:05:33.688] INFO (213): Installed tool poetry in 24.5s.\n"
  "stderr": ""
}

DEBUG: Executing command (branch="renovate/websockets-12.x-lockfile")
{
  "command": "poetry update --lock --no-interaction websockets"
}

DEBUG: exec completed (branch="renovate/websockets-12.x-lockfile")
{
  "durationMs": 3301
  "stdout": "Updating dependencies\nResolving dependencies...\n"
  "stderr": "Creating virtualenv poetry-fails-websocket-3AHzs_c1-py3.12 in /home/ubuntu/.cache/pypoetry/virtualenvs\n"
}

DEBUG: poetry.lock is unchanged (branch="renovate/websockets-12.x-lockfile")
DEBUG: No package files need updating (branch="renovate/websockets-12.x-lockfile")
DEBUG: No updated lock files in branch (branch="renovate/websockets-12.x-lockfile")
DEBUG: No files to commit (branch="renovate/websockets-12.x-lockfile")
DEBUG: Setting current branch to master (branch="renovate/websockets-12.x-lockfile")
DEBUG: latest commit (branch="renovate/websockets-12.x-lockfile")
{
  "branchName": "master"
  "latestCommitDate": "2024-04-23T15:51:35+02:00"
}

DEBUG: getPrCache()
DEBUG: ensureDependencyDashboard()
DEBUG: Ensuring Dependency Dashboard
DEBUG: http cache: saving https://api.github.com/repos/msw-kialo/renovate-poetry-websockets/issues/2 (etag=W/"a8efe6b0f9a1408f4ea13263618ee9b30b8fafa7c860eac2b2f7411e4f52a148", lastModified=Tue, 23 Apr 2024 14:04:14 GMT)
DEBUG: http cache: saving https://api.github.com/repos/msw-kialo/renovate-poetry-websockets/issues/2 (etag=W/"a8efe6b0f9a1408f4ea13263618ee9b30b8fafa7c860eac2b2f7411e4f52a148", lastModified=Tue, 23 Apr 2024 14:04:14 GMT)
DEBUG: ensureIssue(Dependency Dashboard)
DEBUG: http cache: saving https://api.github.com/repos/msw-kialo/renovate-poetry-websockets/issues/2 (etag=W/"a8efe6b0f9a1408f4ea13263618ee9b30b8fafa7c860eac2b2f7411e4f52a148", lastModified=Tue, 23 Apr 2024 14:04:14 GMT)
DEBUG: Patching issue
DEBUG: Issue updated
DEBUG: validateReconfigureBranch()
DEBUG: No reconfigure branch found
DEBUG: Removing any stale branches
DEBUG: config.repoIsOnboarded=true
DEBUG: No renovate branches found
DEBUG: PackageFiles.clear() - Package files deleted
DEBUG: Branch summary
{
  "cacheModified": true
  "baseBranches": [
    {
      "branchName": "master",
      "sha": "7a02e022b84a2cf1c2ce359558d0659700311922"
    }
  ]
  "branches": []
  "defaultBranch": "master"
  "inactiveBranches": [
    "renovate/websockets-12.x-lockfile"
  ]
}

DEBUG: branches info extended
{
  "branchesInformation": [
    {
      "branchName": "renovate/websockets-12.x-lockfile",
      "prNo": null,
      "prTitle": "Update dependency websockets to v12",
      "result": "no-work",
      "upgrades": [
        {
          "datasource": "pypi",
          "depName": "websockets",
          "displayPending": "",
          "fixedVersion": "11.0.3",
          "currentVersion": "11.0.3",
          "currentValue": "^11.0.3",
          "newValue": "^11.0.3",
          "newVersion": "12.0",
          "packageFile": "pyproject.toml",
          "updateType": "major",
          "packageName": "websockets"
        }
      ]
    }
  ]
}

DEBUG: Renovate repository PR statistics
{
  "stats": {
    "total": 0,
    "open": 0,
    "closed": 0,
    "merged": 0
  }
}

[github-actions[bot]]

Hi there,

This issue or discussion is missing some logs, making it difficult or impossible to help you. Depending on which situation applies follow one, some or all of these instructions.

No logs at all

If you haven't posted any log yet, we need you to find and copy/paste the log into the issue template.

Finding logs on hosted app

Select me to read instructions

If you use the Mend Renovate app (GitHub):

1. Log in to the Mend Developer Portal
2. Navigate to the correct organization and repository
3. Locate the appropriate log (it may not always be the latest one)
4. Copy/paste the log contents
5. Follow the steps in the formatting your logs section

Finding logs when self-hosting

Select me to read instructions

Read the Renovate docs, troubleshooting, self-hosted to learn how to find the logs.

Insufficient logs

Select me to read instructions

If you already gave us a log, and the Renovate team said it's not enough, then follow the instructions from the No logs at all section.

Formatting your logs

Select me to read instructions

Please put your logs in a <details> and <summary> element like this:

<details><summary>Select me to see logs</summary>

```
Copy/paste your logs here, between the starting and ending backticks
```

</details>

If you feel the logs are too large to paste here, please use a service like GitHub Gist and paste the link here.

Good luck,

The Renovate team

[rarkins]

Please include packageFiles with updates log for this dependency plus the full branch log

[msw-kialo]

I updated the log in the original discussion.

[rarkins]

It's probably related to #28514

i.e. Renovate's poetry versioning is not reliable if non-semver versions are combined with semver-like syntax such as ^ ranges.

In this case it might be possible to work around this limitation easier than the other case with truly non-semver values.

[rarkins]

It's possible it might start working again as soon as there's a semver release, such as 12.0.1

[rarkins]

Note: this could likely be reproduced with a unit test for poetry versioning's getNewValue(), does not need a repo, or to be a lock file update specifically

[msw-kialo]

  ● modules/versioning/poetry/index › getNewValue("^1.0.3", "bump", "1.0.3", "12.0") === "^12.0"

    expect(received).toEqual(expected) // deep equality

    Expected: "^12.0"
    Received: "^1.0.3"

      246 |         newVersion,
      247 |       });
    > 248 |       expect(res).toEqual(expected);
          |                   ^
      249 |     },
      250 |   );
      251 |

      at lib/modules/versioning/poetry/index.spec.ts:248:19

Correct, I will try to contribute a PR with a fix.