Lerna 2022 Plans

[vsavkin]

We will be publishing a more detailed roadmap later. This is a short version briefly describing the main priorities for this year.

Maintenance

Our goal is to keep Lerna well-maintained: remove all deprecations and security vulnerabilities, set up some good issue and PR management rules, do regular triage, and do regular release planning. We want to make sure Lerna is in good shape such that OSS projects and enterprises can rely on it.

Robust e2e tests

We are going to set up Verdaccio-based e2e-tests. It's the only way to make sure all combinations of npm/pnpm/yarn and linux/windows/mac work well. We have done this before. The result should be better e2e tests that are identical to the actual usage. This should completely eliminate the need to manually test releases, which is currently required.

Bug fixes

We will prioritize the bugs and will start working our way through the high-priority ones.

Docs/website

Currently, the website is outdated; the docs are too brief and not as easy to consume compared to similar tools. We are going to address it.

Fast task execution

Currently, many teams are moving away from Lerna because its task orchestrator (using p-map and p-queue) is relatively slow and doesn't scale well. For instance, most large projects cannot use "lerna run" and "lerna exec". Our goal is to address this, and make Lerna's executor as fast as any other tool.

Summary

Our two goals are:

• Make Lerna more robust: improve maintenance, add better e2e tests, fix bugs.
• Modernize Lerna: update its docs so they are just as good as other tools', update its task executor so it is just as fast as others'. There is some perception that Lerna is something folks are used to using and new projects should look elsewhere. Our goal is to change this.

[LayZeeDK]

End-to-end tests

What do you require from Verdaccio for end-to-end tests? Did you consider using GitHub Packages?

[AgentEnder]

Verdaccio based tests allow publishing the packages to an ephemeral local registry. This means you can publish the current state of master (or a PR) before running the tests, and then can install packages with your regular package manager during the tests.

Unless I'm missing something, GitHub packages doesn't support short-lived releases, and IIRC there is an upper limit on the number of versions you can release.

[LayZeeDK]

We can release the package under a unique name in the format <github-organization>/<any-package-name> and delete it afterward using one of the GitHub APIs, for example the REST API https://docs.github.com/en/rest/packages

Unlimited storage and bandwidth for public GitHub Packages.

There's a 500 MB storage limit on private GitHub Packages for the Free tier https://docs.github.com/en/billing/managing-billing-for-github-packages/about-billing-for-github-packages#about-billing-for-github-packages

[juanpicado]

My opinion might be a bit biased but there I go. You would lose ability to run E2E tests locally (which I guess a contributor would be happy to be able to) aside other minor benefits and potential service outages issues.

[MasterKale]

Lerna has a great NPM publishing capability that I'd hate to see get deprioritized over the desire to improve its build orchestration capabilities. Is it too early in the handoff to seek commitment to maintaining and/or improving this capability as well?

[AgentEnder]

Lerna's publishing capabilities are one of the main features lacking inside of Nx, and is definitely something we are committed to maintaining and improving over time

Checkout the main Nx repo (https://github.com/nrwl/nx), where you'll see that we just swapped out our publishing setup to use lerna. Using it for other projects that we remain committed to will help ensure that it's maintained and improved as we encounter pain points outside of reported issues.

[jeffbcross]

+1 what @AgentEnder said, Lerna's publishing was one of the big opportunities we saw for synergy with Nx. Publishing support was the main reason we reached out to @evocateur about taking over Lerna and to continue evolving its publishing capabilities to cover Nx's use cases instead of reinventing the wheel. TBH, the task orchestration work was easier than we expected, with big performance and DX impact, which is why we tackled it so early after taking over.

[jeffbcross]

To be clear, the task orchestration was easier than expected because we've been working on it for 5 years with Nx, and could integrate Nx under the covers pretty easily. Not because task orchestration is an easy problem in general 😅

[JamesHenry]

@MasterKale just to offer a concrete follow up here, please check out our freshly published Lerna 7 Roadmap: #3410

It covers key improvements we are looking to make to versioning and publishing capabilities in Lerna over the next 6 months

[EWhite613]

PNPM support

Is there any plan to support pnpm? There was a discussion going on #1818 But the deadness of Lerna had some people just moving on from lerna and to pnpm and changesets

[EWhite613]

Yarn v2/v3 support

The same is relevant to yarn v2/v3 as well. As documented in #2449

[EWhite613]

Posting here for more visibility

From slack(some context in there about current plan) via @JamesHenry:

We will next be working on some dedicated guides for using lerna with npm, yarn and pnpm, and we will fix any issues with the various package managers as we encounter them

If you are running into specific issues, please make sure they have been reported on the lerna repo by either upvoting/commenting on an existing one or opening a new one yourself with as much information as possible

Thanks @JamesHenry

[alexn-s]

awesome. looking forward to use lerna with pnpm.

[JamesHenry]

Just for completeness here, we added pnpm support a little while ago in v5

[ScottAwesome]

First and foremost, thanks for taking over this project! I think it could make itself a real big comeback under this stewardship model.

Lerna 5 is looking very interesting to me! I like its publishing capabilities, but I really want to leverage it holistically, but have found that it does not yet support pnpm or yarn 3, to this end, I know its coming, however, I have a question about lerna bootstrap --hoist in particular:

One thing that pnpm provides with its symlinked module structrue (if you don't use shamefully-hoist) is that it isolates your modules well, so there is no phantom dependency problem.

npm will be supporting this soon

and yarn already does via the pnpm option

So my question is this, with the fact that these isolate modes are coming (already landed in the two big package managers, yarn and pnpm, and will eventually land in npm, by version 9 I imagine?) what is the future of lerna bootstrap in this world?

One thing I liked about lerna bootstrap, is if you were creative, you could before the bootstrap process, read your package.json files and extract all the dependency entries, and pass them as --nohoist arguments. This allowed you to hoist your CLI tools but not your actual dependencies, and it worked amazingly well (at least for me, we're talking 40 packages and probably 10K files maybe?) but over time yarn 1 got long in the tooth and the npm support has never been (and I'll be honest, still isn't, which isn't lerna's fault, it doesn't automatically share a lockfile when in workspace mode) so I switched to pnpm.

However, pnpm has a problem, in that its public-hoist-pattern setting doesn't always hoist the entire dependency tree of the listed dependencies. You have to name the sub dependencies too. lerna bootstrap when invoked in the way I described, didn't have this issue, and it was great.

All of this long winded talk to ask this question: do you think that a good path forward for lerna bootstrap --hoist would be in default to hoisting only devDependencies that are not also linked in any sub package dependencies entry, unless configured to do so (and hopefully on a per package basis). This would solve a ton of problems with tool sharing.

The reality here is, while I appreciate nx, its API is both steep and complex and sometimes, there are simpler tools that can be used / configured and launched via npm scripts that do a more than sufficient job, even at scale, when combined with nx running capabilities and caching (I really hope you guys come up with your own first class watch mode though, thats an area that kind of hurts right now)

Sorry, I know this is a huge winded thing, however I been neck deep in monorepo tools for the last month and half trying to figure out which way my company will go for the foreseeable future and I have to say I think nx forced the least compromises on me but what I'm describing above with nx + lerna in this way would be so killer soo....sorry not sorry 😅

[ScottAwesome]

tl;dr, if lerna bootstrap just symlinked .bin entries so we can run scripts via scripts entries it'd be super cool as an option. Also can we just use lerna bootstrap --hoist to only hoist devDependencies via a config param? That would in the gaps with pnpm, yarn, and npm

Heck if you're open to this idea just tell me where to look and I'll open a PR.

I think things being an augment to workspace management rather than driving the workspace like lerna did for so many years (and rightly) is the best new approach, since all 3 of the major package managers take workspaces seriously now, its really about filling in the gaps maybe?

[evocateur]

The future of lerna bootstrap should be deprecation, in my opinion.

Lerna is not a package manager, and lerna bootstrap was always intended as a temporary workaround until package managers gained the capability to manage workspaces themselves. Package managers will always be better at managing packages than Lerna.

[rxgx]

Unfortunately having Lerna manage dependencies of the inner packages has led to a lot of confusion in our org. Most problems come from users trying to run install, ls, or audit within the root of the inner packages rather than the project's root (source of hoist truth).

[JamesHenry]

I can confirm our official endorsement for what @evocateur (as the previous lead maintainer of Lerna) has been encouraging folks for years at this point - please look to move away from lerna boostrap and other legacy package management features which are better handled by npm, yarn and pnpm.

I have expanded upon this view at length in our freshly published Lerna 7 Roadmap, please take a look at what our plans are for lerna boostrap: #3410

[joebowbeer]

I would like to see more support and better documentation for using Lerna to publish from automated CI/CD pipelines.

The from-package positional argument was added to the publish cmd for use in pipelines that do not have the capability to push commits ("what you code-review is what you publish").

This is an important use case that is hidden in the documentation.

ICYMI - I saw a related PR (#3097) to add from-package to the bump cmd, to support automated semver increments, I think.

[JamesHenry]

Thanks @joebowbeer! This is great feedback, we will definitely keep building out the docs

[catdadcode]

Is the ultimate goal to integrate Lerna and Nx? I'm confused about which one to pick and what the ultimate future of both is now that they are under the same umbrella. I've read that Lerna can now use Nx for tasks and Nx now uses Lerna for publishing, etc. etc. It kinda seems to me like they are going to merge eventually? Or am I off base here?

[alexn-s]

i think a bit more information on this topic would be helpful. i stayed with lerna since pnpm support arrived, but tried nx again and noticed quite some overlap (imo).

[JamesHenry]

We are fully committed to both projects! Please check out the lerna content on https://blog.nrwl.io as well as the recently published talks from Nx Conf, especially:

https://www.youtube.com/watch?v=nVYprf3-5GM and https://www.youtube.com/watch?v=CNdDv2MsBuw

When we can leverage Nx implementation details (which we have spent years refining and testing) to benefit Lerna users, without disrupting them unnecessarily, we will, as we have proven successfully with lerna run.

[JamesHenry]

Hi Folks, thanks a lot for the great feedback and discussion on here, please note I have just published our Lerna v7 Roadmap here as a fresh discussion: #3410