repository.secondary=true + dependency.source broken?

[pierresouchay]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: MAC OS

• Poetry version: 1.1.12

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/pierresouchay/60f3e8b9e20ee0d23f0e4604c7a73019

Issue

Following a discussion on #4920

When creating a source called "my-source" with secondary=true, I was expecting the repository to be hit under those conditions:

• if a given package is not present on pypi
• source=my-source is set on a dependency

This is actually not the case, the repository my-source is actually hit each time for every package, for instance when doing a poetry update.

This is causing a few issues if "my-source" is very slow for instance, as time increase on my local setup from 5s to 120s (a project with lots of dependencies).

Is there is a way to use "my-source" ONLY for a few packages?

Here is the output of poetry update -vvv with the pyproject.toml from https://gist.github.com/pierresouchay/60f3e8b9e20ee0d23f0e4604c7a73019 👍

poetry update -vvv
Using virtualenv: <REDACTED>
Updating dependencies
Resolving dependencies...
   1: fact: multi-sources is 0.1.0
   1: derived: multi-sources
   1: fact: multi-sources depends on requests (^2.27.1)
   1: fact: multi-sources depends on changelog (^0.5.7)
   1: selecting multi-sources (0.1.0)
   1: derived: changelog (>=0.5.7,<0.6.0)
   1: derived: requests (>=2.27.1,<3.0.0)
my-source: 1 packages found for changelog >=0.5.7,<0.6.0
   1: selecting changelog (0.5.7)
my-source: 1 packages found for requests >=2.27.1,<3.0.0
   1: fact: requests (2.27.1) depends on urllib3 (>=1.21.1,<1.27)
   1: fact: requests (2.27.1) depends on certifi (>=2017.4.17)
   1: fact: requests (2.27.1) depends on charset-normalizer (>=2.0.0,<2.1.0)
   1: fact: requests (2.27.1) depends on idna (>=2.5,<4)
   1: selecting requests (2.27.1)
   1: derived: idna (>=2.5,<4)
   1: derived: charset-normalizer (>=2.0.0,<2.1.0)
   1: derived: certifi (>=2017.4.17)
   1: derived: urllib3 (>=1.21.1,<1.27)
PyPI: No release information found for certifi-0, skipping
PyPI: 22 packages found for certifi >=2017.4.17
my-source: 22 packages found for certifi >=2017.4.17
   1: selecting certifi (2021.10.8)
PyPI: No release information found for urllib3-0.3, skipping
PyPI: No release information found for urllib3-0.3.1, skipping
PyPI: No release information found for urllib3-0.4.0, skipping
PyPI: No release information found for urllib3-0.4.1, skipping
PyPI: 28 packages found for urllib3 >=1.21.1,<1.27
my-source: 28 packages found for urllib3 >=1.21.1,<1.27
   1: selecting urllib3 (1.26.8)
PyPI: No release information found for idna-0.1, skipping
PyPI: 10 packages found for idna >=2.5,<4
my-source: 10 packages found for idna >=2.5,<4
   1: selecting idna (3.3)
PyPI: 11 packages found for charset-normalizer >=2.0.0,<2.1.0
my-source: 11 packages found for charset-normalizer >=2.0.0,<2.1.0
   1: selecting charset-normalizer (2.0.10)
   1: Version solving took 0.189 seconds.
   1: Tried 1 solutions.

Finding the necessary packages for the current system

No dependencies to install or update

In this example, I used:

• secondary = true for "my-source"

The dependencies:

• requests = { version = "^2.27.1" }
• changelog = {version = "^0.5.7", source = "my-source"}

As shown, all packages are checked on my-source, while I expect only changelog to be checked against this repo.

If my source takes 1 sec to respond, the update process takes a very long time... Is it working as intended, but if so, could it be explained better or is it a bug?

If working as intended, would it be possible to use a source ONLY for the packages I want by default?

[thoward-godaddy]

I see this as well, for my case I only add a secondary source for publishing. No deps come from the secondary source when building. However, poetry update takes about 2 minutes to run compared to 2 seconds when using pypi.

[flecno]

I observe the same and wondered that the secondary = true option has no effect. Based on the access logs of my private repo I see that all packages are requested there and not from PyPi first. I have used poetry version 1.1.12 and 1.1.13

[abn]

This is expected behavior. See https://python-poetry.org/docs/master/repositories/#secondary-package-sources.

[pierresouchay]

This is expected behavior. See https://python-poetry.org/docs/master/repositories/#secondary-package-sources.

Well ok, but what is the alternative as the performance is very poor?

Create a pypi primary source as default, mark as defaut, specify all packages being checked against the one explicitly and only for the 1-2 packages being secondary, specify the secondary?

The last time I checked I could not do that, because perf of legacy repos was poor and I could not add pypy as a non-legacy...

@abn what is your suggestion?

[abn]

poetry source add --secondary foo url
poetry add pycowsay --source=pypi
poetry add secret-package --source=foo

This should work out of the box. Note you don't have to add PyPI as a source. But I suspect requests for transient dependencies will still query both repositories at present.

For more context see #3855 (comment)

#5442 should improve this situation. But the real fix for avoiding all requests will be a new feature to specify a source as something that is queried for explicit packages only. But this introduces a different challenge for transient dependencies.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.