PDM lock file update fails due to missing authentication

[pineapple-pokopo]

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us what version of Renovate you run.

37.189.2

If you're self-hosting Renovate, select which platform you are using.

GitLab self-hosted

What is your question?

I use PDM for Python package management and Google Artifact Registry (GAR) as a private registry and PyPi mirror. I managed to get Renovate to create MRs by setting the relevant hostRule to authenticate to GAR.

Unfortunately, the separate CI job that runs pdm update --no-sync <package> to update the pdm.lock file fails because it cannot authenticate to GAR. Authentication to GAR can be done using the keyring and keyrings.google-artifactregistry-auth packages but these are not available in the environment where pdm update runs.

The only workaround I see, is to create a custom Renovate image that includes all relevant Python/PDM versions and keyring and keyrings.google-artifactregistry-auth pre-installed, but of course this comes with maintenance costs.

Are there any other solutions that do not require a custom Renovate image?

Logs (if relevant)

Logs

Command failed: pdm update --no-sync <package>
WARNING: Project requires a python version of >=3.11,<3.12, The virtualenv is being created for you as it cannot be matched to the right version.
INFO: python.use_venv is on, creating a virtualenv for this project...
Virtualenv is created successfully at /builds/<dir>/.venv
STATUS: Resolving dependencies
See /tmp/pdm-lock-ygbd2l2l.log for detailed debug log.
[PdmException]: The credentials for europe-west4-python.pkg.dev are not provided. To give them via interactive shell, please rerun the command with `-v` option.
WARNING: Add '-v' to see the detailed traceback

[rarkins]

Are you sure that keyring is the only way to provide credentials? e.g. this looks like an alternative: https://github.com/pdm-project/pdm/blob/ddd26878f2a8c32a63ede330d44a45934cbf5742/docs/docs/usage/config.md?plain=1#L210-L211

[pineapple-pokopo]

Hi @rarkins, thanks for the swift response! Keyring is indeed not the only way to set credentials but I failed to make it work using other methods. What I thought could work:

• Set the following env vars in the Gitlab CI job that runs renovate:
  • PDM_PYPI_USERNAME=oauth2accesstoken
  • PDM_PYPI_PASSWORD=$(gcloud auth print-access-token) <-- this works as I added gcloud CLI to the Renovate image and my Gitlab runners are on Google Cloud Platform
• However, these env vars do not seem to get propagated to the process that runs pdm update? I guess this would require setting these env vars explicitly, as has been done for Poetry:

  renovate/lib/modules/manager/poetry/artifacts.ts

  Lines 128 to 135 in 7a57d88

  	if (matchingHostRule.username) {
  	envVars[`POETRY_HTTP_BASIC_${formattedSourceName}_USERNAME`] =
  	matchingHostRule.username;
  	}
  	if (matchingHostRule.password) {
  	envVars[`POETRY_HTTP_BASIC_${formattedSourceName}_PASSWORD`] =
  	matchingHostRule.password;
  	}

Another option would be to configure credentials in the PDM config file, but I am not sure where to store this? Ideally, I would want to configure this globally for Renovate and not in each repo where I use PDM.

Moreover, the above only holds for a single index that is called pypi. Using additional indexes with custom names would require setting credentials for those as well. It seems this cannot be done through env vars but only in the PDM config file or in pyproject.toml. Hence, I think I could not create a global config for Renovate only (also because I don't know all custom index names beforehand).

[rarkins]

You can try setting exposeAllEnv in your bot/admin config (not renovate.json in repo) to true so that all env is passed to child processes.

[pineapple-pokopo]

Thanks, that worked! For reference, I used customEnvVariables to not expose all env vars to child processes:

  customEnvVariables: {
    "PDM_PYPI_USERNAME": "oauth2accesstoken",
    "PDM_PYPI_PASSWORD": process.env.ACCESS_TOKEN <-- set beforehand by Gitlab CI job
  },

For the case where pypi is the only package index that is used, this is a good solution as it does not need any dependencies. However, this will not work for custom package indices and their credentials would need to be specified otherwise. I will try to add keyring and keyrings.google-artifactregistry-auth to the Renovate image and see if that works.

[rarkins]

I also created this feature request to populate such variables automatically from hostRules: #27491

[pineapple-pokopo]

Thanks!

I tried adding keyring and keyrings.google-artifactregistry-auth to the Renovate image and I think this should work. However, there's an issue using a system-installed copy of keyring in combination with PDM, see frostming/unearth#102.

[pineapple-pokopo]

Update: frostming/unearth#102 was fixed, so having keyring and keyrings.google-artifactregistry-auth available globally should enable authentication without needing to set PDM_PYPI_PASSWORD. Note that a username is still needed (oauth2accesstoken) otherwise PDM defaults to __token__ (this is due to a limitation in keyring, see jaraco/keyring#430).