feat: binarySource=install

[rarkins]

Changes:

Adds binarySource=install. Supports npm, composer, jb.

Context:

Closes #11984

Documentation (please check one with an [x])

• I have updated the documentation, or
• No documentation update is required

How I've tested my work (please tick one)

I have verified these changes via:

• Code inspection only, or
• Newly added/modified unit tests, or
• No unit tests but ran on a real repository, or
• Both unit tests + ran on a real repository

[viceice]

LGTM from code, waiting for docs review. 😉

[rarkins]

@viceice testing this locally with:

docker run --rm -e GITHUB_RENOVATE_TESTS_TOKEN -e LOG_LEVEL=debug -w `pwd` -v `pwd`:`pwd` renovate/renovate yarn start renovate-reproductions/12958

Getting error:

DEBUG: Executing command (repository=renovate-reproductions/12958, branch=renovate/eslint-8.x)
       "command": [
         "install-tool npm 8.2.0",
         "hash -d npm",
         "npm install --package-lock-only --no-audit --ignore-scripts"
       ]
DEBUG: rawExec err (repository=renovate-reproductions/12958, branch=renovate/eslint-8.x)
       "err": {
         "killed": false,
         "code": 1,
         "signal": null,
         "cmd": "install-tool npm 8.2.0",
         "stdout": "",
         "stderr": "No USER_NAME defined - skipping: \n",
         "message": "Command failed: install-tool npm 8.2.0\nNo USER_NAME defined - skipping: \n",
         "stack": "Error: Command failed: install-tool npm 8.2.0\nNo USER_NAME defined - skipping: \n\n    at ChildProcess.exithandler (child_process.js:383:12)\n    at ChildProcess.emit (events.js:400:28)\n    at ChildProcess.emit (domain.js:475:12)\n    at maybeClose (internal/child_process.js:1058:16)\n    at Socket.<anonymous> (internal/child_process.js:443:11)\n    at Socket.emit (events.js:400:28)\n    at Socket.emit (domain.js:475:12)\n    at Pipe.<anonymous> (net.js:686:12)"
       }

[viceice]

Renovate isn't passing that env. Need to think about how to fallback. 🤔

[rarkins]

Renovate isn't passing that env. Need to think about how to fallback. 🤔

Feel free to push to this branch if any changes are needed here. Sounds like a buildpack change though?

[viceice]

Yes, needs some buildpack changes. I don't want to force renovate to pass env, which is basically already known

[HonkingGoose]

I think we should hyphenate third-party tools.

[rarkins]

@viceice I specified USER_NAME and then got this next error:

DEBUG: lock file error (repository=renovate-reproductions/12958, branch=renovate/eslint-8.x)
       "err": {
         "killed": false,
         "code": 243,
         "signal": null,
         "cmd": "install-tool npm 8.2.0",
         "stdout": "Installing tool npm v8.2.0
",
         "stderr": "npm notice 
npm notice New major version of npm available! 7.24.2 -> 8.2.0
npm notice Changelog: <https://**redacted**@8.2.0` to update!
npm notice 
npm ERR! code EACCES
npm ERR! syscall rename
npm ERR! path /usr/local/node/14.18.2/lib/node_modules/npm
npm ERR! dest /usr/local/node/14.18.2/lib/node_modules/.npm-1QkxvXuY
npm ERR! errno -13
npm ERR! Error: EACCES: permission denied, rename '/usr/local/node/14.18.2/lib/node_modules/npm' -> '/usr/local/node/14.18.2/lib/node_modules/.npm-1QkxvXuY'
npm ERR!  [Error: EACCES: permission denied, rename '/usr/local/node/14.18.2/lib/node_modules/npm' -> '/usr/local/node/14.18.2/lib/node_modules/.npm-1QkxvXuY'] {
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'rename',
npm ERR!   path: '/usr/local/node/14.18.2/lib/node_modules/npm',
npm ERR!   dest: '/usr/local/node/14.18.2/lib/node_modules/.npm-1QkxvXuY'
npm ERR! }
npm ERR! 
npm ERR! The operation was rejected by your operating system.
npm ERR! It is likely you do not have the permissions to access this file as the current user
npm ERR! 
npm ERR! If you believe this might be a permissions issue, please double-check the
npm ERR! permissions of the file and its containing directories, or try running
npm ERR! the command again as root/Administrator.

[viceice]

😕 I know the error. Need to fix at buildpack.

[viceice]

@rarkins can you test agasin with pulling latest renovate image? it has now some changes, so it should work as expected now.

[rarkins]

New problem which is confusing to me. Running:

docker run --rm -e GITHUB_RENOVATE_TESTS_TOKEN -e LOG_LEVEL=debug -w `pwd` -v `pwd`:`pwd` renovate/renovate yarn start renovate-reproductions/12958

with binarySource=install in config.js produces an error:

DEBUG: exec completed (repository=renovate-reproductions/12958, branch=renovate/eslint-8.x)
       "cmd": "npm install --package-lock-only --no-audit --ignore-scripts",
       "durationMs": 1653,
       "stdout": "",
       "stderr": "(node:170) UnhandledPromiseRejectionWarning: Error [ERR_REQUIRE_ESM]: Must use import to load ES Module: /Users/rhys/src/renovatebot/renovate/node_modules/npm/node_modules/chalk/source/index.js
require() of ES modules is not supported.
require() of /Users/rhys/src/renovatebot/renovate/node_modules/npm/node_modules/chalk/source/index.js from /Users/rhys/src/renovatebot/renovate/node_modules/npm/lib/utils/explain-dep.js is an ES module file as it is a .js file whose nearest parent package.json contains "type": "module" which defines all .js files in that package scope as ES modules.
Instead rename index.js to end in .cjs, change the requiring code to use import(), or remove "type": "module" from /Users/rhys/src/renovatebot/renovate/node_modules/npm/node_modules/chalk/package.json.

    at new NodeError (internal/errors.js:322:7)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1102:13)
    at Module.load (internal/modules/cjs/loader.js:950:32)
    at Function.Module._load (internal/modules/cjs/loader.js:790:12)
    at Module.require (internal/modules/cjs/loader.js:974:19)
    at require (internal/modules/cjs/helpers.js:93:18)
    at Object.<anonymous> (/Users/rhys/src/renovatebot/renovate/node_modules/npm/lib/utils/explain-dep.js:1:15)
    at Module._compile (internal/modules/cjs/loader.js:1085:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
    at Module.load (internal/modules/cjs/loader.js:950:32)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:170) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 1)
(node:170) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.

When running "locally" (mac) with binarySource=docker, I instead get:

DEBUG: exec completed (repository=renovate-reproductions/12958, branch=renovate/eslint-8.x)
       "cmd": "docker run --rm --name=renovate_node --label=renovate_child -v "/tmp/renovate/repos/github/renovate-reproductions/12958":"/tmp/renovate/repos/github/renovate-reproductions/12958" -v "/tmp/renovate/cache":"/tmp/renovate/cache" -e NPM_CONFIG_CACHE -e npm_config_store -e ARTIFACTORY_TOKEN -w "/tmp/renovate/repos/github/renovate-reproductions/12958" docker.io/renovate/node bash -l -c "install-tool npm 8.2.0 && hash -d npm 2>/dev/null || true && npm install --package-lock-only --no-audit --ignore-scripts"",
       "durationMs": 11936,
       "stdout": "Installing tool npm v8.2.0
/home/ubuntu/.npm-global/bin/npm -> /home/ubuntu/.npm-global/lib/node_modules/npm/bin/npm-cli.js
/home/ubuntu/.npm-global/bin/npx -> /home/ubuntu/.npm-global/lib/node_modules/npm/bin/npx-cli.js
+ npm@8.2.0
added 214 packages from 96 contributors in 7.359s
8.2.0

up to date in 939ms

13 packages are looking for funding
  run `npm fund` for details
",
       "stderr": "npm WARN using --force Recommended protections disabled.

[viceice]

The problem is yarn start, it's prepending local ./node_modules/.bin to path. So local npm will be executed. I'ts because of semantic-release:

> yarn why npm
yarn why v1.22.17
[1/4] Why do we have the module "npm"...?
[2/4] Initialising dependency graph...
[3/4] Finding dependency...
[4/4] Calculating file sizes...
=> Found "npm@7.24.2"
info Reasons this module exists
   - "semantic-release#@semantic-release#npm" depends on it
   - Hoisted from "semantic-release#@semantic-release#npm#npm"
info Disk size without dependencies: "17.75MB"
info Disk size with unique dependencies: "26.93MB"
info Disk size with transitive dependencies: "32.93MB"
info Number of shared dependencies: 140
Done in 1.22s.

[viceice]

I think we need to remove local node_modules from PATH when calling child tools, so we force to use the global commands. 🤔

You can run yarn run env to see all env from yarn`.

Or we simply remove the bin link at postinstall, so local npm executable isn't visible in path`

[rarkins]

Blocked by containerbase/base#220 but otherwise seems to be working.

This test repo has three PRs, which were created with npm 6, then 8, then 6: https://github.com/renovate-reproductions/12961/pulls?q=is%3Apr+is%3Aopen+sort%3Aupdated-desc

Install times for npm are slower than I'd like - about 30 seconds - but I'm assuming that's npm's fault and not ours. The third PR using the same npm 6 as the first one takes only 1 second to return from child process, as expected.

[rarkins]

@viceice I think this is ready for final review/merge now

[renovate-release]

🎉 This PR is included in version 30.2.0 🎉

The release is available on:

• GitHub release
• 30.2.0

Your semantic-release bot 📦🚀