PDM authenticate to private nexus repository

[jackson-chris]

What would you like help with?

I would like help with my configuration

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Github using full image 37.351.2-full

Please tell us more about your question or problem

I am using pdm to manage dependencies. Those dependencies are being pulled from a private nexus repo. Renovatebot is failing to update the pdm.lock file as it cannot authenticate with the repository. Error shown below. I have the following relevant configuration:

      "secrets": {
        "PDM_PYPI_USERNAME": "process.env.PDM_PYPI_USERNAME",
        "PDM_PYPI_PASSWORD": "process.env.PDM_PYPI_PASSWORD"
      },
      "customEnvVariables": {
        "CI": "true",
        "PDM_PYPI_USERNAME": "{{ secrets.PDM_PYPI_USERNAME }}",
        "PDM_PYPI_PASSWORD": "{{ secrets.PDM_PYPI_PASSWORD }}"
      },

I am also setting these env variables at the renovatebot process level, the first two are for automatic host rule creation, the latter two are to be referenced as secrets and injected into subprocess for lock updates:

PYPI_NEXUS_MYCOMPANY_COM_USERNAME
PYPI_NEXUS_MYCOMPANY_COM_PASSWORD
PDM_PYPI_USERNAME
PDM_PYPI_PASSWORD

NOTE these have equivalent values PDM_PYPI_USERNAME=PYPI_NEXUS_MYCOMPANY_COM_USERNAME and PDM_PYPI_PASSWORD=PYPI_NEXUS_MYCOMPANY_COM_PASSWORD.

My pyproject.toml has the following section:

[[tool.pdm.source]]
name = "internal"
url = "https://nexus.mycompany.com/repository/mycompany-myteam-pypi/simple"
verify_ssl = true
include_packages = ["myteam-db-model"]
exclude_packages = ["*"]

Usually when building locally we include a pdm.toml file with the following content:

[pypi.internal]
username = "REDACTED"
password = "REDACTED"

we obviously don't check this file into source control nor would we want to.

What am I doing wrong? What's the correct way to inject PDM credentials for renovate to consume?

Logs (if relevant)

Logs

DEBUG: Failed to update PDM lock file (repository=myorg/myrepo, baseBranch=main, branch=renovate-ab6c6a725789c02)
       "err": {
         "cmd": "/bin/sh -c pdm update --no-sync django-storages",
         "stderr": "STATUS: Resolving dependencies\nSee /home/ubuntu/.local/state/pdm/log/pdm-lock-rn6njci_.log for detailed debug log.\n[PdmException]: The credentials for nexus.mycompany.com are not provided. To give them via interactive shell, please rerun the command with `-v` option.\nWARNING: Add '-v' to see the detailed traceback\n",
         "stdout": "Updating packages: django-storages.\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/myorg/myrepo",
           "encoding": "utf-8",
           "env": {
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/containerbase/tools/python/3.12.3/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/containerbase/tools/python/3.12.3/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/containerbase/tools/python/3.12.3/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "CI": "true",
             "PDM_PYPI_USERNAME": "**redacted**",
             "PDM_PYPI_PASSWORD": "**redacted**",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: pdm update --no-sync django-storages\nSTATUS: Resolving dependencies\nSee /home/ubuntu/.local/state/pdm/log/pdm-lock-rn6njci_.log for detailed debug log.\n[PdmException]: The credentials for nexus.mycompany.com are not provided. To give them via interactive shell, please rerun the command with `-v` option.\nWARNING: Add '-v' to see the detailed traceback\n",
         "stack": "ExecError: Command failed: pdm update --no-sync django-storages\nSTATUS: Resolving dependencies\nSee /home/ubuntu/.local/state/pdm/log/pdm-lock-rn6njci_.log for detailed debug log.\n[PdmException]: The credentials for nexus.mycompany.com are not provided. To give them via interactive shell, please rerun the command with `-v` option.\nWARNING: Add '-v' to see the detailed traceback\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:529:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:292:12)"
       }

[rarkins]

Where are you configuring those "secrets"? E.g. in a config.js file?

You can't quote "process.env.FOO". That's just a string. You need to remove the ""

[jackson-chris]

The secrets are configured in a base.json file and that file is referenced via RENOVATE_CONFIG_FILE env variable. For self hosted can I update RENOVATE_CONFIG_FILE to point to a base.js file instead, where the content would be:

module.exports = {
  <<contents of base.json>>
};

at the same time removing the quotes around the process.env.FOO ref?

[rarkins]

JSON files cannot read from the environment. And strings are strings if you have quotation marks

[jackson-chris]

So it sounds like I can make the change I suggested to get this working indicated by your thumbs up. Can you confirm whether or not host rule from environment work with PDM?

For example should having these env set:

PYPI_NEXUS_MYCOMPANY_COM_USERNAME
PYPI_NEXUS_MYCOMPANY_COM_PASSWORD

supplied credentials for PDM updating binaries pulled from nexus.mycompany.com? Or if not with the PYPI_ prefix what is the appropriate way via host rules from environment?

I saw this #27491 issue logged but not sure if that was identify the same gap or not?

[jackson-chris]

So it sounds like I can make the change I suggested to get this working indicated by your thumbs up.

Unfortunately that didn't work either. It still complaining the credentials are missing. I tried setting env variables like so in my base.js file:

    module.exports = {
      secrets: {
        PYPI_NEXUS_MYCOMPANY_COM_USERNAME: process.env.PYPI_NEXUS_MYCOMPANY_COM_USERNAME,
        PYPI_NEXUS_MYCOMPANY_COM_PASSWORD: process.env.PYPI_NEXUS_MYCOMPANY_COM_PASSWORD
      },
      customEnvVariables: {
        CI: "true",
        PDM_PYPI_USERNAME: "{{ secrets.PYPI_NEXUS_MYCOMPANY_COM_USERNAME }}",
        PDM_PYPI_PASSWORD: "{{ secrets.PYPI_NEXUS_MYCOMPANY_COM_PASSWORD }}",
        PDM_INTERNAL_USERNAME: "{{ secrets.PYPI_NEXUS_MYCOMPANY_COM_USERNAME }}",
        PDM_INTERNAL_PASSWORD: "{{ secrets.PYPI_NEXUS_MYCOMPANY_COM_PASSWORD }}"
      },

I tried both PYPI and INTERNAL as I wasn't sure if PDM uses the [pypi.internal] ref to pick env variables for credentials.

[rarkins]

You're going to need to provide more information than "didn't work"

[jackson-chris]

It appears that it's not using those env variables when processing the lock file update as its still logging the same error I already provided:

DEBUG: Failed to update PDM lock file (repository=myorg/myrepo, baseBranch=main, branch=renovate-ab6c6a725789c02)
       "err": {
         "cmd": "/bin/sh -c pdm update --no-sync django-storages",
         "stderr": "STATUS: Resolving dependencies\nSee /home/ubuntu/.local/state/pdm/log/pdm-lock-rn6njci_.log for detailed debug log.\n[PdmException]: The credentials for nexus.mycompany.com are not provided. To give them via interactive shell, please rerun the command with `-v` option.\nWARNING: Add '-v' to see the detailed traceback\n",
         "stdout": "Updating packages: django-storages.\n",
         "options": {
           "cwd": "/tmp/renovate/repos/github/myorg/myrepo",
           "encoding": "utf-8",
           "env": {
             "HOME": "/home/ubuntu",
             "PATH": "/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/containerbase/tools/python/3.12.3/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/containerbase/tools/python/3.12.3/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/.cargo/bin:/home/ubuntu/.local/bin:/go/bin:/home/ubuntu/bin:/opt/containerbase/tools/python/3.12.3/bin:/home/ubuntu/.npm-global/bin:/home/ubuntu/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
             "LC_ALL": "C.UTF-8",
             "LANG": "C.UTF-8",
             "CI": "true",
             "PDM_PYPI_USERNAME": "**redacted**",
             "PDM_PYPI_PASSWORD": "**redacted**",
             "PDM_INTERNAL_USERNAME": "**redacted**",
             "PDM_INTERNAL_PASSWORD": "**redacted**",
             "CONTAINERBASE_CACHE_DIR": "/tmp/renovate/cache/containerbase"
           },
           "maxBuffer": 10485760,
           "timeout": 900000
         },
         "exitCode": 1,
         "name": "ExecError",
         "message": "Command failed: pdm update --no-sync django-storages\nSTATUS: Resolving dependencies\nSee /home/ubuntu/.local/state/pdm/log/pdm-lock-rn6njci_.log for detailed debug log.\n[PdmException]: The credentials for nexus.mycompany.com are not provided. To give them via interactive shell, please rerun the command with `-v` option.\nWARNING: Add '-v' to see the detailed traceback\n",
         "stack": "ExecError: Command failed: pdm update --no-sync django-storages\nSTATUS: Resolving dependencies\nSee /home/ubuntu/.local/state/pdm/log/pdm-lock-rn6njci_.log for detailed debug log.\n[PdmException]: The credentials for nexus.mycompany.com are not provided. To give them via interactive shell, please rerun the command with `-v` option.\nWARNING: Add '-v' to see the detailed traceback\n\n    at ChildProcess.<anonymous> (/usr/local/renovate/lib/util/exec/common.ts:99:11)\n    at ChildProcess.emit (node:events:529:35)\n    at ChildProcess.emit (node:domain:489:12)\n    at Process.ChildProcess._handle.onexit (node:internal/child_process:292:12)"
       }

[rarkins]

The error shows that matching env variables are being passed through