Poetry selects wrong Python version if Poetry's installation differs from python3 (or python3 changes)

[ShadowLNC]

• Poetry version: 1.2.2
• Python version: CPython 3.9.12
• OS version and name: MacOS (darwin) 12.6.1 Monterey (ARM architecture)
• pyproject.toml: See below

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate. Related but not the same as far as I can tell:
  Poetry not using the correct python version #7075
  Poetry does not set the path for or activate the virtual environment if the active python version differs from that used to install poetry #6978
  poetry env info has wrong virtualenv info #6893
• I have consulted the FAQ and blog for any relevant entries or release notes.
• (N/A) If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

If Poetry's internal virtual environment (used to execute Poetry) doesn't match the version resolved by the python3 command, then Poetry doesn't select the correct Python version (defined in pyproject.toml) for installation when creating a new virtual environment for a project.

There are two different ways that I've found to achieve this scenario.

Scenario 1:

1. Install Python 3.10 (or 3.9 or 3.8, just referring to 3.10 from now on) from https://python.org (using standard installer)
2. (Now $PATH resolves python3 to 3.10)
3. Run curl -sSL https://install.python-poetry.org | python3 - as officially supported installation mechanism
4. Install Python 3.11 from https://python.org
5. (Now $PATH resolves python3 to 3.11)

Scenario 2:

1. Install Python 3.9 (or any version other than 3.11) from https://python.org (using standard installer)
2. Install Python 3.11 from https://python.org
3. (Now $PATH resolves python3 to 3.11)
4. Run curl -sSL https://install.python-poetry.org | python3.9 -

Outcome

Relaunch the terminal to ensure $PATH changes take effect.

$ poetry install --no-root
The currently activated Python version 3.9.12 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.0)
Creating virtualenv poetrytest in /Users/scott/Documents/automation/poetrytest/.venv
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing attrs (22.1.0)

This all looks correct, but then I activate the virtual environment with source .venv/bin/activate.

Expected

The shell prompt shows (poetrytest-py3.11) and python --version shows 3.11.0. I would also expect poetry shell to work in place of source .venv/bin/activate.

Actual

The shell prompt shows (poetrytest-py3.11) and python --version shows 3.9.12 (or whatever older version I used).

$ poetry shell

Current Python version (3.9.12) is not allowed by the project (~3.11).
Please change python executable via the "env use" command.

I've had a colleague verify this behaviour on their MacOS machine as well, and another colleague indicated that it may work properly on Ubuntu (but I have not verified that first hand).

Files

poetry.toml

[virtualenvs]
in-project = true

pyproject.toml (minimal packages just to show installation)

[tool.poetry]
name = "poetrytest"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.11"
attrs = "^22.1.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.lock

[[package]]
name = "attrs"
version = "22.1.0"
description = "Classes Without Boilerplate"
category = "main"
optional = false
python-versions = ">=3.5"

[package.extras]
dev = ["cloudpickle", "coverage[toml] (>=5.0.2)", "furo", "hypothesis", "mypy (>=0.900,!=0.940)", "pre-commit", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "sphinx", "sphinx-notfound-page", "zope.interface"]
docs = ["furo", "sphinx", "sphinx-notfound-page", "zope.interface"]
tests = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "zope.interface"]
tests-no-zope = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins"]

[metadata]
lock-version = "1.1"
python-versions = "~3.11"
content-hash = "088f78f8dc08e7c9c2577c94e3669da3a7f2e3a6a2c4caa83caa754e5c626bef"

[metadata.files]
attrs = [
    {file = "attrs-22.1.0-py2.py3-none-any.whl", hash = "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"},
    {file = "attrs-22.1.0.tar.gz", hash = "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6"},
]

[neersighted]

This is partially by design; you must set https://python-poetry.org/docs/configuration/#virtualenvsprefer-active-python-experimental if you want Poetry to resolve Python instead of using sys.interpreter. However, there are bugs related to our auto-discovery of a compatible Python; @finswimmer put some work in here recently, I'll leave it to him to determine if this is fixed in 1.3 or there is something novel here.

[ShadowLNC]

Thanks for the quick reply 😄

If I add prefer-active-python = true in poetry.toml, I get this:

$ poetry install --no-root
/bin/sh: python: command not found

Command python -c "import sys; print(sys.executable)" errored with the following return code 127, and output: 

Python 2 was removed from newer MacOS, but I wouldn't expect Poetry to be looking for something other than Python 3.

I'm not sure how this relates to prefer-active-python though? Poetry tells me it's installing 3.11 (which it has discovered, and which I requested in pyproject.toml), but ends up installing 3.9 (which I think matches sys.interpreter).

[finswimmer]

Hey,

unfortunately I'm not able to reproduce it. Would be helpful if you or anyone else can provide a reproducer with docker.

fin swimmer

[ShadowLNC]

Edited to add: Since Poetry 1.3.0 has been released, these Ubuntu results are on Poetry 1.3.0.

Hi - I think I have managed to get a set of reproduction steps in docker. Ubuntu locks the python3 symlink to 3.10, so I've changed it such that we install Poetry against 3.9, and request 3.10 in pyproject.toml.

I created poetry-repro-setup.sh:

apt-get update -y
# Need software-properties-common for apt-add-repository
apt-get install -y software-properties-common python3.10 python3.10-venv python3.11 python3.11-venv curl
add-apt-repository -y ppa:deadsnakes/ppa
apt-get install -y python3.9 python3.9-venv  # From deadsnakes

mkdir poetrytest
cd poetrytest

cat - > poetry.toml <<EOF
[virtualenvs]
in-project = true
EOF

cat - > pyproject.toml <<EOF
[tool.poetry]
name = "poetrytest"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "~3.10"
attrs = "^22.1.0"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
EOF

cat - > poetry.lock <<EOF
[[package]]
name = "attrs"
version = "22.1.0"
description = "Classes Without Boilerplate"
category = "main"
optional = false
python-versions = ">=3.5"

[package.extras]
dev = ["cloudpickle", "coverage[toml] (>=5.0.2)", "furo", "hypothesis", "mypy (>=0.900,!=0.940)", "pre-commit", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "sphinx", "sphinx-notfound-page", "zope.interface"]
docs = ["furo", "sphinx", "sphinx-notfound-page", "zope.interface"]
tests = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins", "zope.interface"]
tests-no-zope = ["cloudpickle", "coverage[toml] (>=5.0.2)", "hypothesis", "mypy (>=0.900,!=0.940)", "pympler", "pytest (>=4.3.0)", "pytest-mypy-plugins"]

[metadata]
lock-version = "1.1"
python-versions = "~3.10"
content-hash = "f7ca16790b2695440ca3017a7bb30137759c6612e03aec49ae380742bbc484ed"

[metadata.files]
attrs = [
    {file = "attrs-22.1.0-py2.py3-none-any.whl", hash = "sha256:86efa402f67bf2df34f51a335487cf46b1ec130d02b8d39fd248abfd30da551c"},
    {file = "attrs-22.1.0.tar.gz", hash = "sha256:29adc2665447e5191d0e7c568fde78b21f9672d344281d0c6e1ab085429b22b6"},
]
EOF

I ran the docker container as follows:

docker run --rm -it -v $(pwd)/poetry-repro-setup.sh:/poetry-repro-setup.sh ubuntu bash

Then, within the docker container:

chmod +x poetry-repro-setup.sh
/poetry-repro-setup.sh
curl -sSL https://install.python-poetry.org | python3.9 - 
export PATH="/root/.local/bin:$PATH"
cd /poetrytest

And then when I try to install the virtual environment, I get the same behaviour as MacOS:

root@84b3855a09a8:/poetrytest# poetry install --no-root
The currently activated Python version 3.9.16 is not supported by the project (~3.10).
Trying to find and use a compatible version. 
Using python3 (3.10.6)
Creating virtualenv poetrytest in /poetrytest/.venv
Installing dependencies from lock file

Package operations: 1 install, 0 updates, 0 removals

  • Installing attrs (22.1.0)
root@84b3855a09a8:/poetrytest# poetry shell

Current Python version (3.9.16) is not allowed by the project (~3.10).
Please change python executable via the "env use" command.
root@84b3855a09a8:/poetrytest# source .venv/bin/activate
(poetrytest-py3.10) root@84b3855a09a8:/poetrytest# python --version
Python 3.9.16

If, instead of installing Poetry with 3.9, I install it with 3.11 (as provided in the container via above script - an older 3.11.0rc1 version from the official Ubuntu Jammy repository, instead of deadsnakes), I get different behaviour.

• If 3.10 (pointed at by the python3 symlink) is requested by pyproject.toml and poetry.lock, it warns that 3.11 is active, finds 3.10 (Using python3 (3.10.6) is output), and installs correctly.
• If 3.11 is requested by pyproject.toml and poetry.lock (with the content hash from my original post, 088f78f8dc08e7c9c2577c94e3669da3a7f2e3a6a2c4caa83caa754e5c626bef), it does not warn (as 3.11 is active and I am requesting 3.11), but installs 3.10 instead, with the 3.11 venv prompt.

Here's my full results (where python3 always refers to python3.10 on Ubuntu).
Edited to add: Since Poetry 1.3.0 has been released, these Ubuntu results are on Poetry 1.3.0.

Poetry Installed With	Requested Python	Virtualenv Prompt	Installed Python
3.9	3.9	3.9	✅ 3.9
3.9	3.10	3.10	❌ 3.9
3.9	3.11	3.11	✅ 3.11
3.10	3.9	3.9	✅ 3.9
3.10	3.10	3.10	✅ 3.10
3.10	3.11	3.11	✅ 3.11
3.11	3.9	3.9	✅ 3.9
3.11	3.10	3.10	✅ 3.10
3.11	3.11	3.11	❌ 3.10

(For completeness, the poetry.lock content hash for Python 3.9 is 1e20473c3d8816fdca442b8208dd4268efad46603021991e7219def665cf6772.)

[finswimmer]

Thanks a lot for your excellent reproducer @ShadowLNC. 👍 I can reproduce it with your instructions.

Hopefully I have some time the next days to find out what's going on.

[finswimmer]

I think I found the place where things go wrong.

Poetry finds python3 as a suitable executable, because it's version is 3.10.6. Now python3 is passed as an argument to virtualenv. virtualenv runs in Poetry's own venv, so python3 points to Python 3.9.

What we must do here is to always pass the full path to the executable. (This is what already happening if virtualenvs.prefer-active-python is set to true.)

[finswimmer]

Fix is prepared: #7221

[rpf3]

@finswimmer will this be released soon? I just upgraded to a new system and am getting bit by this issue using Python 3.10.6 as well.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.