Poetry ignores platform constraint when using local path

[M-J-Murray]

• 18.04.2 Ubuntu 5.0.0-32-generic:
• Poetry version 1.0.10:
• Link to toml https://gist.github.com/M-J-Murray/7c3d538db07413614d9ff6f04ce28b57:

Issue

Hello,
I'm attempting to setup poetry such that a couple of local libraries are only installed when using windows. However, the platform argument is failing in two ways. When I use platform and path arguments on ubuntu, like so:

gdal = {platform = "windows", path = "thirdparty/GDAL-3.1.2-cp37-cp37m-win32.whl"}

Even though I am not on windows, poetry keeps trying to install the library and failing because the install will only work on windows. As a workarounds I have tried:

gdal = {markers = "sys_platform == 'windows'", path = "thirdparty/GDAL-3.1.2-cp37-cp37m-win32.whl"}
gdal = {markers = "sys_platform != 'windows'", path = "thirdparty/GDAL-3.1.2-cp37-cp37m-win32.whl"}
gdal = {markers = "sys_platform == 'windows' and sys_platform != 'linux'", path = "thirdparty/GDAL-3.1.2-cp37-cp37m-win32.whl"}

But it keeps ignoring my platform requirement. I have a feeling the problem comes from the combination of path and platform markers.

[abn]

@M-J-Murray can you try this with the current pre-release or master? Attempted to reproduce this with master and was unable.

[M-J-Murray]

@abn I installed poetry using: curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python.
Does this not install from master?

[abn]

Sorry for the delay in response. That installs the latest available release. Which right now is 1.1.2. You should try this version.

[espdev]

I have the same problem with multiple constraints.
There is a part of my pyproject.toml:

[tool.poetry.dependencies]
python = ">=3.8,<3.9"

numpy = [
    { version = "^1.19.4", platform = "linux" },
    { path = "wheels/numpy-1.19.4+mkl-cp38-cp38-win_amd64.whl", platform = "windows" },
]

On Linux I get an error:

Updating dependencies
Resolving dependencies... (4.0s)

  SolverProblemError

  Because poetry-lock depends on numpy (1.19.4+mkl wheels/numpy-1.19.4+mkl-cp38-cp38-win_amd64.whl) which doesn't match any versions, version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

On Windows it works, but created lock file will be broken on Linux platform:

[[package]]
name = "numpy"
version = "1.19.4"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = false
python-versions = ">=3.6"

[[package]]
name = "numpy"
version = "1.19.4+mkl"
description = "NumPy is the fundamental package for array computing with Python."
category = "main"
optional = false
python-versions = ">=3.6"

[package.source]
type = "file"
url = "wheels/numpy-1.19.4+mkl-cp38-cp38-win_amd64.whl"

[metadata]
lock-version = "1.1"
python-versions = ">=3.8,<3.9"
content-hash = "00a3930fc10a275304528a51ea4a43e855e2b04780aaaae0e1464f05680e3f89"

I tried to install the same project on WSL in the same project directory with the lock file created on Windows. Poetry is trying to install win_amd64 wheel file on Linux:

Creating virtualenv poetry-lock-MNvJcrA9-py3.8 in /home/esp/.cache/pypoetry/virtualenvs
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing numpy (1.19.4+mkl /mnt/c/Temp/poetry_lock/wheels/numpy-1.19.4+mkl-cp38-cp38-win_amd64.whl): Failed

  EnvCommandError

  Command ['/home/esp/.cache/pypoetry/virtualenvs/poetry-lock-MNvJcrA9-py3.8/bin/pip', 'install', '--no-deps', '/mnt/c/Temp/poetry_lock/wheels/numpy-1.19.4+mkl-cp38-cp38-win_amd64.whl'] errored with the following return code 1, and output:
  ERROR: numpy-1.19.4+mkl-cp38-cp38-win_amd64.whl is not a supported wheel on this platform.
  WARNING: You are using pip version 20.2.2; however, version 20.3.3 is available.
  You should consider upgrading via the '/home/esp/.cache/pypoetry/virtualenvs/poetry-lock-MNvJcrA9-py3.8/bin/python -m pip install --upgrade pip' command.

  at ~/.poetry/lib/poetry/utils/env.py:1074 in _run
      1070│                 output = subprocess.check_output(
      1071│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1072│                 )
      1073│         except CalledProcessError as e:
    → 1074│             raise EnvCommandError(e, input=input_)
      1075│
      1076│         return decode(output)
      1077│
      1078│     def execute(self, bin, *args, **kwargs):

It seems to me that this because the lock file does not contain any info about platform marker. The lock file is not cross-platform without platform markers.

I'm using Poetry 1.1.4

[SamuelBradley]

This is still an issue with poetry 1.1.7

[emptonaut]

This issue appears to still exist on 1.1.11.

[john-sandall]

This is still happening on 1.2.1

I'm trying to have Poetry lock a single poetry.lock that can install pytorch on both macOS and Linux environment. I don't want to use secondary source as this dramatically slows down all poetry lock or poetry add by scanning the secondary source for every package in my deps, and I can't link to the remote wheel URL because Poetry doesn't cache the file and every poetry lock then redownloads a 2GB pytorch wheel (also slow).

Using a local wheel solves this neatly, but then we run into this bug. My current solution isn't nice, but it works at least: pre-commit executes a Python script immediately after poetry lock that finds and then inserts environment marker lines into poetry.lock for torch and torchvision: https://gist.github.com/john-sandall/05f03d6e39978c1aef29e00dd61e253a

Maybe this will help someone, or help to create a reproducible example?

[neersighted]

@radoering any chance you want to take a whack at this? Looks like another issue around markers and multiple constraint dependencies when using direct origin deps.

[radoering]

I can't reproduce this yet. In general, the markers are not required in the lockfile to choose the correct version for installation.

@john-sandall Can you share your pyproject.toml and poetry.lock? (That should be more helpful to reproduce the issue.)

By the way, I noticed that some posts in this issue declare a platform constraint that doesn't make sense. On Windows sys_platform normally is win32. (Can be verified with import sys; print(sys.platform).) That's why something like platform = "windows" or markers = "sys_platform == 'windows'" is not fulfilled even on Windows and something like markers = "sys_platform != 'windows'" is always fulfilled (even on Windows).

[billfreeman44]

I am running into this on Windows 11. Building our docker image was building wheels for hdbscan so we built it ourselves. It should fall to 0.8.28 on windows, and that works (poetry run pip install hbdscan)

This is what our pyproject.toml looks like.

[[tool.poetry.dependencies.hdbscan]]
python = "^3.9"
markers = "platform_python_implementation == 'CPython' and sys_platform == 'linux' and platform_machine == 'x86_64'"
url = "https://github.com/rxfork/hdbscan/releases/download/0.8.28/hdbscan-0.8.28-cp39-cp39-linux_x86_64.whl"

[[tool.poetry.dependencies.hdbscan]]
python = "^3.9"
markers = "platform_python_implementation == 'CPython' and sys_platform == 'linux' and platform_machine == 'aarch64'"
url = "https://github.com/rxfork/hdbscan/releases/download/0.8.28/hdbscan-0.8.28-cp39-cp39-linux_aarch64.whl"

[[tool.poetry.dependencies.hdbscan]]
version = "0.8.28"

I verified with import sys; print(sys.platform) that my system is win32. Would this logic help reproduce @radoering ?

[neersighted]

@billfreeman44 your issue is different -- this issue is specifically for path dependencies, not URL dependencies. That being said, the following works as expected on every platform for me -- I suspect your markers are just slightly off:

[[tool.poetry.dependencies.hdbscan]]
python = "3.9"
markers = "platform_python_implementation == 'CPython' and sys_platform == 'linux' and platform_machine == 'x86_64'"
url = "https://github.com/rxfork/hdbscan/releases/download/0.8.28/hdbscan-0.8.28-cp39-cp39-linux_x86_64.whl"

[[tool.poetry.dependencies.hdbscan]]
python = "3.9"
markers = "platform_python_implementation == 'CPython' and sys_platform == 'linux' and platform_machine == 'aarch64'"
url = "https://github.com/rxfork/hdbscan/releases/download/0.8.28/hdbscan-0.8.28-cp39-cp39-linux_aarch64.whl"

[[tool.poetry.dependencies.hdbscan]]
markers = "python_version != '3.9' or sys_platform != 'linux'"
version = "0.8.28"

If you need more help or advice, please start a Discussion, join Discord, or create a detailed issue report if you think there may be a bug.

[dimbleby]

poetry considers that if a dependency doesn't specify its source, then any package with a compatible version is a candidate to meet that dependency, regardless of that package's source.

So at install time on linux: poetry sees the dependency on numpy 1.19.4, and considers the package that points at the local wheel to be a perfectly valid choice. In effect the pyproject.toml had overlapping requirements for numpy, even though that wasn't what was intended.

(In fact I also don't reproduce the reported problem today, but I reckon that's just because of a coincidence about the order in which poetry happens to store packages in the lockfile rather than because any actual fix has happened).

So to deal with this you can explicitly specify the pypi source on the linux numpy dependency. That un-overlaps the constraints, they are now unambigously pointing at different things. Then at install time, poetry will be able to tell that the file package is not compatible with the pypi requirement, and do the right thing.

Except that per #6710 this doesn't quite work today. Still, in principle that's a sensible answer - and reduces this one to being a duplicate of that one.

[dimbleby]

To clarify, I was analysing this comment as being the same as #6710 The original is long fixed, if indeed it ever reproduced in the first place.

[neersighted]

Thanks for the analysis @dimbleby -- we've solved the original issue and the issue that I replied to. Your conclusions track for me and answer the version @espdev commented on. Closing as resolved/tracking @espdev's version in #6710.

[frankblubaugh]

I'm still having issues with this limitation. I'm linking to different paths on a shared drive that are mapped based on the base operating system and folder shares for the compute resources. These are compiled libraries wrapped using maturin.

my_library = [{file="<path_1_on_host_1.whl>",markers="sys_platform == 'darwin'",platform="macos"}, {file="<path_2_on_host_2.whl>",markers="sys_platform != 'darwin'",platform="linux"}]

When I try to resolve poetry lock on host_1 I keep getting the error "path_2_on_host_2 does not exist" I don't understand why poetry would be trying to resolve path_2 when my host platform is a Mac not Linux. There isn't the cloned repository path as @dimbleby suggests could be the issue as this is a self-contained library.

If I manually update the path on each host individually every time it works just fine, but I don't see why I should need to do that if there is the platform flag

[mpizenberg]

I can confirm that I’m also having issue with a combination of markers and path. I have the following dependencies in my pyproject.toml:

[tool.poetry.dependencies]
python = "3.8.*"
ids_peak = [
  {markers = "platform_system == 'Windows'", path = "C:/Program Files/IDS/ids_peak/generic_sdk/api/binding/python/wheel/x86_64/ids_peak-1.5.0.0-cp38-cp38-win_amd64.whl"},
  {markers = "platform_system == 'Linux'", path = "vendor/ids-peak_2.3.0.0-15823_amd64/local/share/ids/bindings/python/wheel/ids_peak-1.5.0.0-cp38-cp38-linux_x86_64.whl"},
]

Where ids_peak is a library to control cameras provided by the manufacturer. I’ve tried both variants with platform_system and sys_platform (in that case with win32 and linux) and in all cases the locking of dependencies fails. In linux (wsl) it complains about the path to the windows dep, and in windows it complains about the path to the linux dep.

> poetry lock
Path /home/matthieu/git/ids-peak-python/C:/Program Files/IDS/ids_peak/generic_sdk/api/binding/python/wheel/x86_64/ids_peak-1.5.0.0-cp38-cp38-win_amd64.whl for ids_peak does not exist
Updating dependencies
Resolving dependencies... (0.0s)

Path /home/matthieu/git/ids-peak-python/C:/Program Files/IDS/ids_peak/generic_sdk/api/binding/python/wheel/x86_64/ids_peak-1.5.0.0-cp38-cp38-win_amd64.whl for ids_peak does not exist

Poetry version: 1.4.1

[hyliu1989]

@neersighted Could you look into this issue again since you closed it? The issue still pops up when I have a specification exactly like this. The pyproject.toml and lock file are provided.

The attached lock file was generated in a Windows machine. Regardless of whether this file exists or not, poetry lock command in MacOS failed at missing path

Path /Users/.../folder_containing_this_pyproject_toml/C:/python_venv/src/opencv-python-cuda for opencv-contrib-python does not exist

[tool.poetry]
name = "deep_learning"
version = "0.1.0"
description = "Deep learning root"
authors = ["John Doe <john.doe@gmail.com>"]

[tool.poetry.dependencies]
python = ">=3.10, <3.11"
# 1st command: poetry add --source pytorch-gpu-src torch torchvision torchaudio  # Hand adding platform.
torch = [
    {platform = "win32", version = "^2.0.1+cu118", source = "pytorch-gpu-src"},
    {platform = "darwin", version = "^2.0.1"},
]
torchvision = [
    {platform = "win32", version = "^0.15.2+cu118", source = "pytorch-gpu-src"},
    {platform = "darwin", version = "^0.15.2"},
]
torchaudio = [
    {platform = "win32", version = "^2.0.2+cu118", source = "pytorch-gpu-src"},
    {platform = "darwin", version = "^2.0.2"},
]
# 2nd command: poetry add tensorflow-io-gcs-filesystem==0.31.0
tensorflow-io-gcs-filesystem = {version = "0.31.0"}
# 3rd command: poetry add numpy=^1.22 scipy matplotlib jupyter notebook pandas scikit-learn tensorflow seaborn nltk keras ipywidgets pooch ipympl
numpy = "^1.22"
scipy = "^1.11.1"
matplotlib = "^3.7.2"
jupyter = "^1.0.0"
notebook = "^7.0.6"
pandas = "^2.0.3"
scikit-learn = "^1.3.0"
# opencv-python = "^4.8.0.74"
tensorflow = "^2.13.0"
seaborn = "^0.12.2"
nltk = "^3.8.1"
keras = "^2.13.1"
ipywidgets = "^8.1.1"
pooch = "^1.8.0"
ipympl = "^0.9.3"
# 4th command: poetry run pip install tensorflow==2.13.0 (since 2.13.0 was not installed properly)
# 5th command: poetry add --source pytorch-gpu-src torchtext (I got an error on torchtext installation but torchdata 0.6.1 was installed)
pytest = "^7.4.3"

[tool.poetry.group.dev.dependencies]

[tool.poetry.group.cv-compile]
optional = true

[tool.poetry.group.cv-compile.dependencies]
# 6th command: edit this toml for the following local opencv-contrib-python.
opencv-contrib-python = [
    {platform = "win32", path = "C:/python_venv/src/opencv-python-cuda/", develop = false},
    {platform = "darwin", version = "^4.7.0"},
]
#opencv-contrib-python = { git = "git@github.com:hyliu1989/opencv-python-cuda.git", rev = "29fcb4f1f01ae3145f3c1c62485ca91dc3d3adad", develop = false }

[tool.poetry.scripts]
# jnserver = "jupyter notebook"  # jupyter notebook --ip=0.0.0.0

[[tool.poetry.source]]  # added by `poetry source add --priority=explicit pytorch-gpu-src https://download.pytorch.org/whl/cu118`
name = "pytorch-gpu-src"
url = "https://download.pytorch.org/whl/cu118"
priority = "explicit"

[build-system]
requires = [
    "poetry-core>=1.0.0",  # A Rust compiler has to be installed outside of Poetry and is a requirement.
]
build-backend = "poetry.core.masonry.api"

poetry.lock.txt

Updated: the poetry version I used was

# poetry --version
Poetry (version 1.7.0)

[hyliu1989]

Just to update, my reported case above happened in version 1.7.0 of Poetry.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.