How to add encrypted env variable used with yarn 4

[ced-mos]

What would you like help with?

I would like help with my configuration

How are you running Renovate?

Self-hosted

If you're self-hosting Renovate, tell us which platform (GitHub, GitLab, etc) and which version of Renovate.

Gitlab, Renovate V37.101.0

Please tell us more about your question or problem

Lately we moved your node project to yarn 4 where changes at the configuration had to be done. Now I am not able to make it work with private npm packages and encrypted secrets. From how I understand it what is missing is setting up encrypted env variables, which I was not able to achieve.

To ensure that we don't search for a solution for a wrong approach, here my overall goal:

• Use Yarn 4 (preferably with npmScopes in .yarnrc.yml)
• Store encrypted secrets
• Run Renovate-bot again without failures including updates of the yarn.lock files

At the moment it's not clear to me if this is just a misconfiguration or I am taking the wrong approach. Therefore, could you help on how I can achieve that? In addition I would like to know if there is a solution with renovate-bot version 37.101.0 or if we need to update? This as I remarked, that env variables configuration did not work with 37.101.0, where I had to update to a newer version like 37.232.0 on my local setup. However to change the version I would need to check with the infrastructure team, if this is feasible.

Here a minimal example what until now worked without encrypted env variables:

.yarnrc.yml

nodeLinker: node-modules
yarnPath: .yarn/releases/yarn-4.1.1.cjs

injectEnvironmentFiles:
    # made .env.yarn optional as we were unable to set .env.yarn during renovate-bot run
    - .env.yarn?

npmScopes:
    my-private-org:
        npmAuthToken: '${NPM_AUTH_TOKEN}'
        npmRegistryServer: 'https://gitlab.company.com/api/v4/packages/npm/'

package.json

{
  "name": "demo",
  "version": "0.0.0",
  "main": "server.js",
  "scripts": {
    "test": "mocha --exit tests/*",
    "start": "node server.js"
  },
  "license": "MIT",
  "devDependencies": {
    "mocha": "10.4.0",
    "renovate": "37.272.0",
    "supertest": "6.3.4"
  },
  "dependencies": {
    "@my-private-org/my-private-package": "2.0.1",
    "express": "4.19.0"
  },
  "packageManager": "yarn@4.1.1"
}

renovate.json

{
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": [
        "config:base",
        "group:all"
    ],
    "env":{
        "NPM_AUTH_TOKEN": "valid-token"
    },
    "hostRules": [
        {
            "matchHost": "https://gitlab.company.com",
            "encrypted":{
                "token": "valid-token"
            },
            "description": "NPM Token used for private packages"
        }
    ],
    "packageRules": [
        {
            "groupName": "all non-major dependencies",
            "groupSlug": "all-non-major-patch",
            "matchPackagePatterns": [
                "*"
            ],
            "matchUpdateTypes": [
                "minor",
                "patch"
            ]
        }
    ],
    "schedule": [
        "every weekday"
    ]
}

From the documentation it was not clear to me, if the env variables should work in combination with the "encrypted" object but this is what I tested, which failed:

{
    "$schema": "https://docs.renovatebot.com/renovate-schema.json",
    "extends": [
        "config:base",
        "group:all"
    ],
    "encrypted": {
    "env":{
        "NPM_AUTH_TOKEN": "valid-token"
    },
    ...
    }
}

Already many thank for your help.

Logs (if relevant)

Error content of created issue with title "Action Required: Fix Renovate Configuration".

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Env variable name encrypted is not allowed by this bot's allowedEnv., Invalid env variable value: env.encrypted must be a string.

[viceice]

encrypted is not allowed in env. use host rules instead of environment variable

[ced-mos]

This I have already done but it did not work.

[viceice]

remove the auth line with the env var from yarn config and ensure matchHost from host rule matches your registry url exactly

[ced-mos]

I will need to check it next week when I am back in the office.

[ced-mos]

@viceice This solves the problem with the env variables for the renovate-bot run. However this generates the issue, that yarn does not generate the lock files, as the credentials are not found for the external libraries, when running the renovate-bot. Here the output provided in the generated MR:

➤ YN0000: · Yarn 4.1.1
➤ YN0000: ┌ Resolution step
➤ YN0035: │ @my-private-org/my-private-package@npm:2.0.1: Package not found
➤ YN0035: │   Response Code: 404 (Not Found)
➤ YN0035: │   Request Method: GET
➤ YN0035: │   Request URL: https://gitlab.company.com/api/v4/packages/npm/@my-private-org%2fmy-private-package
➤ YN0000: └ Completed in 0s 417ms
➤ YN0000: · Failed with errors in 0s 424ms

When trying to install locally the packages via yarn install on my machine I get the same error which I expect is the same which is used during the renovate-bot run. The log message does sound logical to me as we don't provide the NPM token anymore for the specific private packages during the yarn install process.

As you suggest to remove the authentication from the .yarnrc.yml file, which did not work for me, do you know if your solution could still work after some tweaks?

If not, do you have another idea to make it work?

[ced-mos]

Already a big thank you for taking the time to assist.

Another think I remarked. In the comment you mentioned that I need to ensure that the matchHost from host rule matches the registry url exactly.

In the yarnrc.yml I hadnpmRegistryServer: 'https://gitlab.cistec.com/api/v4/packages/npm/' where in the renovate.json I had"matchHost": "https://gitlab.cistec.com",. This did not work. Also when changing it to "matchHost": "https://gitlab.cistec.com/api/v4/packages/npm/", it did not work. Hence, what do you mean by matching the urls exactly to ensure that not a typo is leading to my problem?

[github-actions[bot]]

Hi there,

Please do not unnecessarily @ mention maintainers like @rarkins or @viceice. Doing so causes annoying mobile notifications and makes it harder to maintain this repository.

For example, never @ mention a maintainer when you are creating a discussion if your desire is to get attention. This is rude behavior, just like shouting out your coffee order in a Starbucks before it's your turn.

It's OK to comment in an issue or discussion after multiple days or weeks. But please, still don't @ mention people. The maintainers try to answer most discussions, but they can't answer all discussions. If you're still not getting an answer, take a look at the information you've given us and see if you can improve it.

Thanks, the Renovate team

[ced-mos]

Note to myself. Original problem is shown in #14756. However, solutions proposed there did not solve my problem.

[rarkins]

@ced-mos can you clarify if I understand right?

• You want to have a .yarnrc.yml in your repo, which includes scopes/private registry information, but no credentials in git
• You want to configure credentials using Renovate config
• You want Renovate to add them to the .yarnrc.yml prior to calling Yarn?

[ced-mos]

Thank you for your quick response.

The public and private key we have already setup which already worked before with the hosted rules passwords before migrating to yarn 4.

However when trying to add env variables (https://docs.renovatebot.com/configuration-options/#env) encrypted in the renovate.json config I did't find in the docs how to make it work. Hence ha I tried several possibilities like embedding the env section in a encrypted field or only the variable itself, which never worked.
Therefore, do you mean the env variables described in https://docs.renovatebot.com/configuration-options/#env ? If so, how would I need to add them precisely in the renovate.json config encrypted?

Note, the unencrypted env variables I was able to configure including the bot configuration with the allowEnvs. Just to be sure what already works and what not.

[rarkins]

I would have expected this to work:

{
  "env": {
    "encrypted": {
      "NPM_AUTH_TOKEN": "........"
    }
  }
}

The above would work only if the admin had configured allowedEnv accordingly.

[ced-mos]

I just tested it with your suggested version on how to add encrypted env variables which did not work. Here the configuration I added to the renovate.json besides adding the variable to allowedEnv:

"env": {
        "encrypted": {
            "NPM_AUTH_TOKEN": "MY-ENCRYPTED-TOKEN"
        }
    },

Here the error output of renovate (V37.320.0):

Location: renovate.json
Error type: The renovate configuration file contains some invalid settings
Message: Env variable name encrypted is not allowed by this bot's allowedEnv., Invalid env variable value: env.encrypted must be a string.

Hence, is this as expected or a bug? From the documentation (https://docs.renovatebot.com/configuration-options/#encrypted) I would have assumed that this could work but as this use case with encrypted env variables is never stated explicitly I am not sure if this is the case.

[rarkins]

This is a validation bug. Can you reproduce it on github.com for us to debug? The value you encrypt can be a dummy one, or even the encrypted string itself can be dummy.

[ced-mos]

I started a repo with https://github.com/ced-mos/renovate-bot-bug-encrypted-env-variable/tree/main where I used the managed renovate bot runner from mend. The error message (ced-mos/renovate-bot-bug-encrypted-env-variable#1) is the same as I mentioned earlier.

However, I am not sure if this is exactly the same issue as it mentions that the env variable does not exists in the allowedEnv where I indeed was not able to find a configuration for that in the managed mend renovate bot. Hence, is this sufficient for you or do you need more? If so, please elaborate that I can prepare it for you.

[ced-mos]

did you read the following ?

https://docs.renovatebot.com/getting-started/private-packages/#yarn-2

Excuse me for the late response. Yes, I read this section. However I have to admit it's not clear for me what exactly I should do.

In the documentation it's mentioned that npmRegistries and npmScopes is not supported in yarnrc.yml but later it explains it will somehow update it in the yarnrc.yml. Now it's not clear, how I would need to adapt my explained dummy configuration above, more precisely how to adapt the yarnrc.yml and renovate.json that it should work and also if I would need to set up a .npmrc file.

Therefore, could you just explain how I would need to set it up precisely based on my described configuration discussion description including the correct URLs?