lerna bootstrap still symlinking when using --scope or --ignore

[Inlustra]

Expected Behavior

Running lerna bootstrap --ignore={@inlustra/my-lib,}
should not symlink the @inlustra/my-lib package with the rest of the packages in the monorepo

I would expect lerna to install @inlustra/my-lib normally from npm (using regular npm i)

Current Behavior

lerna info version 3.0.0-beta.20

lerna symlinks all of the packages, including the ignored packages but doesn't run npm install in those ignored

Possible Solution

I'm still unsure if this is expected behaviour but the naming is certainly confusing if that's the intended solution

Context

We're currently running packages that can take a little while to build due to the libraries we're using (Thanks ionic 😢) as such we decided we'd split the app into multiple packages to hopefully speed up build time for developers.

The idea being that the developer that is working between 2 packages only needs to symlink those and then we can run a typescript build using lerna exec and watch on those packages separately

The other packages can be installed from the npm repository

[evocateur]

--ignore only stops a given package from being bootstrapped from the "root", it doesn't stop transitive (local) dependencies from linking and building, etc. The only case where lerna bootstrap installs a local name from npm is when the version range is not satisfied by the local version.

[Inlustra]

@evocateur We’ve ended up working around it - but out of curiosity, what’s the use case for such a thing? 😊

I feel like I’m missing something, I guess the naming confused me 😅

[evocateur]

I don’t think filtering bootstrap makes sense at all. Then again, I’ve largely stopped using bootstrap altogether.

…

On May 24, 2018, at 16:32, Inlustra ***@***.***> wrote: @evocateur We’ve ended up working around it - but out of curiosity, what’s the use case for such a thing? 😊 I feel like I’m missing something, I guess the naming confused me 😅 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[ibezkrovnyi]

@evocateur
We have the same issue:

let's pretend we have following packages (actually we have more than 200 packages in monorepo):

• A
• B depends on A
• C depends on B and on A
• D depends on C

Actual behavior

if I want to for example, test project C and its dependents, I should run following commands with adding --include-filtered-dependencies, because even if I'm not interested in time consuming npm run build for B and A, they will be symlinked, which means they will have no /dist folder

lerna bootstrap --scope {,C} --include-filtered-dependencies --include-filtered-dependents
lerna run build --scope {,C} --include-filtered-dependencies --include-filtered-dependents
lerna run test --scope {,C} --include-filtered-dependents

Almost always it means to npm install + npm run build ALL 200 monorepo projects

Desired behavior

if lerna symlinked only --scope / --since packages (real scope), I could run:

lerna bootstrap --scope {,C} --include-filtered-dependents
lerna run build --scope {,C} --include-filtered-dependents
lerna run test --scope {,C} --include-filtered-dependents

--ignore

adding all projects that are not in scope+filter flags seems a little bit too complex: I need to create list of all projects and then I need to substract manually projects returned by lerna list --scope/filter flags and pass the rest of packages to the ignore?

Proposal

if project is not in scope (including filter flags include-filtered-dependents / include-filtered-dependencies), it should not be symlinked to node_modules of its dependents. It may be behind some flag? Something like --symlink-ignore-out-of-scope-projects or --symlink-only-projects-in-scope. As I understand change may be very small if not one line + updating command.js + readme :)

Benefits

No need to npm install/rebuild not changed subprojects (A and B in example), it's completely fine to use their npm versions. It may save up to 99% of time.

@Inlustra may I ask for your workaround if it is possible ofc, please? 😳

[evocateur]

@ibezkrovnyi You shouldn't have 200 packages in a monorepo, maybe? Lerna isn't a replacement for proper separation of concerns.

[JabbyPanda]

@ibezkrovnyi was exaggerating by mentioning possible maintenance of 200 packages in single monorepo, the real number of packages in our monorepo is closer to 20.

[evocateur]

@JabbyPanda Regardless of the actual number, bootstrapping in a monorepo will always incur costs beyond that of a "normal" package repository. This is how lerna bootstrap works, and it isn't likely to change given the existing complexity of the command.

[ibezkrovnyi]

Solution

Option 1 - Forking Lerna repo

In the following line additional check can be introduced:

lerna/core/package-graph/index.js

Line 116 in d6e6a42

if (forceLocal || resolved.fetchSpec === depNode.location || depNode.satisfies(resolved)) {

(see #1766)

Option 2 - using simple script

As projects not in scope should not participate in the build at all, it is ok temporary make their versions incompatible with any dependent project.

Example

projects

• button@1.5.3
• list-chooser (which depends on button@^1.0.0)

we ant to build and release

• list-chooser

so we want the following projects to be not linked and be taken from npm

• button@1.5.3

what we need to do in script

According to:

lerna/core/package-graph/index.js

Line 116 in d6e6a42

if (forceLocal || resolved.fetchSpec === depNode.location || depNode.satisfies(resolved)) {

it is enough to temporary break version of all projects not in scope (button@1.5.3), so algorithm is:
`

const listOfProjectgsInScope = exec lerna list --scope // (or lerna changed) depending on your needs
const listOfProjects = exec lerna list
const listofProjectsNotInScope = listOfProjects - listOfProjectsInScope

for each listOfProjectsNotInScope:
set package.json#version to something like 0.0.0-thisIsWrongVersionMarker

// lerna bootstrap (in package-graph) will not link button project into list-chooser/node_modules/button if version of button in monorepo is not compatible with button@^1.0.0 semver range of list-chooser
lerna bootstrap --scope {,anyscope}

lerna run build
lerna run test
etc

// just before publishing with npm publish
for each listOfProjectsNotInScope:
restore package.json#version by git checkout package.json

// publish
lerna exec -- npm publish // or lerna publish if it works for you
`

@evocateur our team has ~80 react components with very similar configuration (webpack, babel, lint, mocha, etc) and in short perspective after evaluating how monorepo goes we might move non-react-components and libraries (used by react components) into monorepo too (to automate testing of dependent projects, to automate releasing, and ofc centralized tooling of non-react projects is also big win when we manage ~120 non-react projects). We are also investigating idea of creating more libraries of components, decreasing number of atomic projects. Microsoft, Google, Facebook all have monorepo in one or another look with huge number of projects inside one single monorepo, and there are on the internet at least one story/conference talk per each company regarding their monorepos, but ofc you are aware of that.

@JabbyPanda if you want I can show you list of projects we develop/support at my desk

[evocateur]

Ah hell, bootstrap is explicitly filterable, so let's just do it the easy way: overwrite targetGraph when filteredPackages.length !== targetGraph.size.

diff --git a/commands/bootstrap/index.js b/commands/bootstrap/index.js
index 29e2d3a..c4ff75e 100644
--- a/commands/bootstrap/index.js
+++ b/commands/bootstrap/index.js
@@ -106,6 +106,11 @@ class BootstrapCommand extends Command {
     chain = chain.then(() => getFilteredPackages(this.targetGraph, this.execOpts, this.options));
     chain = chain.then(filteredPackages => {
       this.filteredPackages = filteredPackages;
+
+      if (filteredPackages.length !== this.targetGraph.size) {
+        // an explicit --scope, --ignore, or --since should only symlink the targeted packages, no others
+        this.targetGraph = new PackageGraph(filteredPackages, "allDependencies", this.options.forceLocal);
+      }
     });
 
     chain = chain.then(() => {

[evocateur]

v3.10.3 has the fix

[ibezkrovnyi]

@evocateur tested this change, works fine! Thank you ❤️

test details

- /project-A
- /project-B
- /project-C that depends on project-A & project-B

> lerna bootstrap --scope {,project-A,project-C}

/project-C/node_modules/project-A is Junction Point to /project-A
/project-C/node_modules/project-B is Folder (taken from npm)
both project-A & project-C bootstrapped

[lock]

This thread has been automatically locked because there has not been any recent activity after it was closed. Please open a new issue for related bugs.