'lerna bootstrap' fails to symlink binaries which are generated during package build

[rix0rrr]

We are using TypeScript. As part of the TypeScript "build", .js files are generated, some of which are command-line tools.

We are doing our TypeScript -> JavaScript compilation in the NPM "prepare" phase.

However, lerna bootstrap does not symlink the binaries that are generated as part of the prepare phase, because it executes lerna link once before running the prepare phases of all targets, and it makes no effort to link later.

From lerna-debug.log I can tell that it does execute preinstall for every package before symlinking, but the "preinstall" phase happens on every machine that installs the package, so is not the correct phase to do our TypeScript compilation in.

Reproduction:

Directory tree
├── lerna.json
├── package.json
└── packages
    ├── pkga
    │   └── package.json
    └── pkgb
        └── package.json

# pkga/package.json
{
  "name": "pkga",
  "version": "1.0.0",
  "main": "index.js",
  "bin": {
      "executable": "executable"
  },
  "scripts": {
      "prepare": "echo '#!/bin/bash' > executable && echo 'echo it runs' >> executable && chmod +x executable"
  }
}

# pkgb/package.json
{
  "name": "pkgb",
  "version": "1.0.0",
  "main": "index.js",
  "scripts": {
      "prepare": "executable"
  },
  "devDependencies": {
      "pkga": "*"
  }
}

In short: executable is produced by pkga and used in the build of pkgb. This would have worked if pkga was downloaded from NPM, but does not work when the build is managed by Lerna.

Expected Behavior

For the build of package pkgb, it should not matter whether dependency pkga is obtained from NPM by npm install, or provided by lerna as part of a monorepo.

Current Behavior

Right now it does, because the symlinks to executables are evaluated before packages are event built, so binaries generated during the build are missed.

Possible Solution

A simple fix would be to run lerna link again after every package is bootstrapped. This would fix the issue, though it may be bad for performance. If you have enough information in the dependency tree to do selective updates that may be even better.

Steps to Reproduce (for bugs)

See the project layout given above.

$ lerna bootstrap
...
lerna ERR! execute Error occured with 'pkgb' while running 'npm run prepare'
{ Error: Command failed: npm run prepare
sh: 1: executable: not found

Proof that this is the issue, running lerna bootstrap again now that executable already exists during the link phase:

$ lerna bootstrap
...
lerna success Bootstrapped 2 packages

Note that running lerna bootstrap multiple times is not a solution, as we might not know in advance how many times we would need to run it! Depends on the amount of packages and the dependencies among them!

Context

We are writing packages in TypeScript. Some packages are build tools, containing command-line scripts that need to be run during the build of other packages.

We emphatically DO NOT want to run the build by hand and check in the generated .js files.

Your Environment

Executable	Version
lerna --version	2.8.0
npm --version	6.1.0
node --version	v8.9.1

OS	Version
NAME	VERSION
Ubuntu	16.04

[evocateur]

Why not just have the binary files live as raw JS, importing a compiled "implementation" and calling it?

That being said, properly-expressed dependencies between the packages should elicit topologically-sorted execution of the lifecycle scripts during bootstrap. I have no experience with tsc's behaviour in this department, but I can guess it's doing a lexical iteration of package directories and failing due to this mismatch.

[rix0rrr]

Why not just have the binary files live as raw JS, importing a compiled "implementation" and calling it?

Yeah, that's we end up doing to work around this, but I still feel lerna's behavior is wrong in this case.

As mentioned, if the upstream dependency was obtained through npm by using npm install, I could have used the binaries in my prepare script with no problems. However, when I build them at the same time using lerna, the same does not work. I was under the impression that lerna would be a replacement for a workflow where multiple packages together need to be manually npm installed or npm linked all the time.

I have no experience with tsc's behaviour in this department, but I can guess it's doing a lexical iteration of package directories and failing due to this mismatch.

No, every package is compiled individually (by doing npm run prepare) in the right topological order. The fact that source is being generated using tsc is incidental--note that my reproducing example had no mention of tsc in there anywhere.

The problem is that lerna is doing the link action at a time before any package has been built, so it fails to symlink the binaries that will be generated (obviously), and takes to effort to correct that later on.

[evocateur]

The problem is that lerna is doing the link action at a time before any package has been built, so it fails to symlink the binaries that will be generated (obviously), and takes to effort to correct that later on.

My apologies for the incorrect assumptions. The bootstrap algorithm is fiendishly complex, but you’re welcome to propose changes to accomplish your goal.

[jdolle]

I just encountered this bug this past week.
For others struggling, my temporary solution was to delete the node_modules and reinstall dependencies for the dependent package. And because I use yarn workspaces, deleting the node modules and re-yarning doesn't need to redownload anything.

"fixSymlink": "rm -rf ./node_modules && yarn",
"build": "yarn fixSymlink && someBin",

[evocateur]

If you're using yarn workspaces, the bug you're experiencing is in Yarn, not lerna. `lerna bootstrap` just calls `yarn` in that case.

…

On Dec 9, 2018, at 17:02, Jeff Dolle ***@***.***> wrote: I just encountered this bug this past week. For others struggling, my temporary solution was to delete the node_modules and reinstall dependencies for the dependent package. And because I use yarn workspaces, deleting the node modules and re-yarning doesn't need to redownload anything. "fixSymlink": "rm -rf ./node_modules && yarn", "build": "yarn fixSymlink && someBin", — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

[acao]

i switched my project from yarn to npm and now am experiencing this problem.
if you are having this issue with yarn, upgrade to the latest and it should resolve it. still a problem for npm however

[acao]

also, with yarn workspaces, i never needed to use lerna link or bootstrap

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[allanhortle]

I'm having this issue too. One of my packages is building a bin file during install which never gets symlinked into the sibling packages.

[stephenmathieson]

Also seeing this with Lerna version 3.4.3.

[rix0rrr]

I actually think an easy way to resolve this is to create the symlinks in the downstream packages, based on what the { "scripts" } section promises will be the final executables, whether they exist yet or not.