Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build tools: Yarn 2, yarn 3 and NPM 7+ for Monorepo Solution #4737

Closed
awentzel opened this issue May 19, 2021 · 34 comments
Closed

build tools: Yarn 2, yarn 3 and NPM 7+ for Monorepo Solution #4737

awentzel opened this issue May 19, 2021 · 34 comments
Labels
area:dev-ops Pertains to build, CI, and other dev-ops work
Projects

Comments

@awentzel
Copy link
Collaborator

awentzel commented May 19, 2021

Perform an investigation to verify if Yarn 2 and yarn 3 or NPM7, can meet our mono-repository needs as features could have changed since the last time we investigated and we should understand the implications of choosing one of the other and if we can achieve our usual development process without Lerna.

Yarn 2

Version 2 of the popular JavaScript package manager Yarn2 has been here for a year now. Yet try to find information on upgrading from 1.X or how it compares to current versions of npm (especially the 7.x dev release), and you’ll get plenty of well-written but outdated blog posts mainly bemoaning the new direction the Yarn team has taken. On its initial release, taking a wait-and-see approach with Yarn 2 may have been the correct call, but it’s time to reconsider that decision.

Here are three reasons you might have waited to make the switch — and why those reasons are out of date in 2021

  1. Backward Compatibility with node_modules
  • Yarn 2 uses a new Plug’n’Play (PnP) architecture that is a huge departure from how npm projects have always worked. PnP does away with the node_modules/ folder where packages get installed. Unfortunately, this is incompatible with certain JavaScript-based projects that rely on that directory to find packages, namely React Native, Flow, and VSCode Extension packages.

  • However, Yarn 2 now offers an option that copies packages to the node_modules/ folder just like Yarn 1, providing backward compatibility for these projects. It literally requires adding a single line to your new .yarnrc.yml configuration file to turn on this feature.

  1. Difficulty Migrating from Yarn 1.x
  • The PnP incompatibilities that plagued the initial Yarn 2 release — and the seismic shift it created in getting rid of that hefty node_modules/ folder — also meant migration could be a big pain for pre-existing projects. But with node_modules/ support added back in, Yarn 2 now provides a clean migration path. There are a not-insignificant number of steps here, especially if you’re moving to PnP support, but the documentation is clear and easy to follow. It shouldn’t actually be that difficult to undertake.
  1. But My Yarn 1.X Setup Isn’t Broken!
  • As the Yarn 2 documentation mentions:
    Even if you don’t use Plug’n’Play nor plan to use it, your project will still benefit from more stable node_modules layouts, improved performances, improved user experience, active development, etc.
    Sooner or later, you still have to migrate.

  • Yarn 2 migration and documentation: Migration | Yarn - Package Manager

Step by step migration to Yarn 2

Yarn 2 Documentation for Migration

  1. Run npm install -g yarn to update the global yarn version to latest v1
  2. Go into your project directory
  3. Run yarn set version berry to enable v2 (cf Install for more details)
  4. If you used .npmrc or .yarnrc, you'll need to turn them into the new format (see also 1, 2)
  5. Add nodeLinker: node-modules in your .yarnrc.yml file
  6. Commit the changes so far (yarn-X.Y.Z.js, .yarnrc.yml, ...)
  7. Runyarn install to migrate the lockfile
  8. Take a look at this article to see what should be gitignored
  9. Commit everything remaining

Some optional features are available via external plugins:

  1. Run yarn plugin import interactive-tools if you want upgrade-interactive

  2. Run yarn plugin list to see what other official plugins exist and might be useful

  3. Commit the yarn plugins

Yarn 3.0.0 Release

Breaking Changes
  1. Node 10 isn't supported anymore.
  2. Plugins can't access yup anymore (we migrated to Typanion as part of Clipanion v3).
  3. To upgrade workspace-tools, remove it from your .yarnrc.yml, upgrade, then import it back.
  4. The enableImmutableInstalls will now default to true on CI (we still recommend to explicitly use --immutable on the CLI).
  5. You can re-allow mutations by adding YARN_ENABLE_IMMUTABLE_INSTALLS=false in your environment variables.
  6. The initVersion and initLicense configuration options have been removed. initFields should be used instead.
    6.Yarn will now generate .pnp.cjs files (instead of .pnp.js) when using PnP, regardless of what the type field inside the manifest is set to.
    7.The virtual folder (used to disambiguate peer dependencies) got renamed from $$virtual into virtual.
  7. The -a alias flag of yarn workspaces foreach got removed; use -A,--all instead, which is strictly the same.
  8. The old PnPify SDK folder (.vscode/pnpify) won't be cleaned up anymore.
  9. The --skip-builds flag from yarn install got renamed into --mode=skip-build.
  10. The bstatePath configuration option has been removed. The build state (.yarn/build-state.yml) has been moved into the install state (.yarn/install-state.gz)
  11. The cache files need to be regenerated. We had to change their timestamps in order to account for a flaw in the zip spec that was causing problems with some third-party tools.
  12. @yarnpkg/pnpify has been refactored into 3 packages:
  13. @yarnpkg/sdks now contains the Editor SDKs
  14. @yarnpkg/pnpify now contains the PnPify CLI compatibility tool that creates in-memory node_modules
  15. @yarnpkg/nm now contains the node_modules tree builder and hoister
  16. @yarnpkg/plugin-node-modules has been renamed to @yarnpkg/plugin-nm
  17. The --clipanion=definitions commands supported by our CLIs will now expose the definitions on the entry point (rather than on .command)

NPM 7

  • With version 7 of npm they've reduced their dependencies by roughly 54%, while increasing the code test coverage by about 17%. It should also include a performance boost in multiple areas according to their own benchmarks.

  • Npm 7 is now the latest version in the npm registry and therefore default. To install the new version of npm, you'll run the following in your command line interpreter of choice: npm install --global npm@latest

Couple new features from NPM 7

  1. Version 2 of package-lock
  • With the new package-lock.json file we'll unlock the ability to do deterministically reproducible builds. It should now include everything npm needs to install the packages needed. Before npm 7 yarn.lock was ignored by npm, but this is no longer the case. It can now use it to keep itself up to date with the package tree.

  • The new lock file should be backward compatible with users of npm 6. Though, when you run npm install in a project with a version 1 lock file it will replace that file with the new structure. This can be avoided by running npm install --no-save when installing.

  1. Workspaces
  • This is one of the new features that I'm most excited about. It includes a set of features that will make the management of multiple packages a lot better. It lets you handle packages from a singular top-level root package. This has already been possible to do with for example yarn, Lerna, or Pnpm.

  • In order to make npm aware that the current project is a workspace you have to add workspaces key in your package.json. This can be done by adding every single sub-folder or by using a glob, like in the example below.

Automatically installing peer dependencies

  • In versions before npm 7 developers had to install the peer dependencies. Now npm will use a new algorithm to ensure the dependencies are installed properly. If a peer dependency, that is not compatible with the specified one, is installed npm 7 will now block the installation.

Breaking changes

  • Since the new version is considered a major version it'll come with a couple of breaking changes. Here are some:
    You can no longer use require() npm's internal modules. Npm now uses the package. exports field.
    The team has completely rewritten npx to internally use npm exec, the npx CLI will still be available. Some functionality changes are to expect. One is that you'll now be prompted if you try to run a module that is not installed yet.

  • The changes mentioned above regarding peer dependencies might disturb some workflows.
    npm audit has a new output.. npm 6 showed all packages by default when running npm ls. With npm 7 it will only show the top-level packages. Run npm ls --all to mimic the behavior from npm 6.

Node Version

**Node Version 14 recommended**

There are many things to look forward to with this major release. Some of the new features include:

  • The once experimental feature, Diagnostic Reporting, is now stable!
  • V8 has been upgraded to V8 8.1
  • Node Streams have been updated
  • Removal of Experimental Modules Warning

Node version 16 current

  • As is standard in Node.js releases, this version will not be promoted to long-term support (LTS) until October. We need the greater ecosystem to try it out and give the community feedback. This will allow us to address any issues in advance and make sure both the release, the ecosystem, and our customers are ready when it’s promoted.

Here are some of the new features and ongoing work in the 16 release, which include:

  1. Updated Platform support 

  2. V8 JavaScript Engine Version 9 

  3.  N-API Version 8 

  4. New Promises APIs 

  5. Async Local Storage APIs 

Summary

  • Before npm7, it was not possible to handle a large monorepo project with npm alone. That why developers switched to yarn. With NPM7, this is no longer an issue, NPM can do exactly what yarn does. npm v7 arrives with a newer version for the package-lock.json format - allowing to reduce the need to read package.json files and to have enough information to reliably describes the full and precise package tree all by itself. More than that, the resulting package tree using the new lockfile is flattened, and this is crucial to boost the performance.
    Support. We have a partnership with JavaScript Cloud Advocates v-Team at Microsoft, who provide information on what happening on NPM 7, Node Js, and GitHub. If we start using npm 7, we will be getting support from them.
    As GitHub owns NPM 7, the most recent information is posted on GitHub.
    npm 7 is now generally available! | The GitHub Blog

  • Yarn has been a leading Monorepo solution sometimes in combination with Lerna. The early release of Yarn 2 has some incompatibility with Certains JavaScript library like react. Sounds they have provided some fix for it. Here is the official link to Yarn: 2 - Installation | Yarn - Package Manager
    The community of Yarn has stated that Yarn 2 is a little slower than Yarn 1. This could be due to changes in Yarn infrastructure and added functionalities. The release of Yarn 3.0.0 has taken the performance into consideration and reduce configuration during the setup process or migration from the old version of Yarn.

  • Node.js. We will use the recommended version 14. This is to prevent a breaking change in the customer application. If we use the latest version 16. Most customers may not be up to date with their node version.

Features Yarn npm
Installation process To install Yarn, npm has to be installed Installation Docs npm gets installed with the Node.JS solutions automatically Installation Docs
Lock file In order to get consistent installs across machines, Yarn needs more information than the dependencies you configure in your package.json. Yarn needs to store exactly which versions of each dependency were installed. Yarn generates a yarn-lock file and it is auto-generated and should be handled entirely by Yarn. As you add/upgrade/remove dependencies with the Yarn CLI, it will automatically update your yarn.lock file. Do not edit this file directly as it is easy to break something. npm generates a package-lock.json file that is automatically generated for any operations where npm modifies either the node_modules tree, or package.json. It describes the exact tree that was generated, such that subsequent installs are able to generate identical trees, regardless of intermediate dependency updates. This file is intended to be committed into source repositories, and serves various purposes: Describe a single representation of a dependency tree such that teammates, deployments, and continuous integration are guaranteed to install exactly the same dependencies. Provide a facility for users to "time-travel" to previous states of node_modules without having to commit the directory itself. To facilitate greater visibility of tree changes through readable source control diffs. And optimize the installation process by allowing npm to skip repeated metadata resolutions for previously-installed packages.
Output log Output logs are clean, brief, and visually distinct Output logs are voluminous
Global dependencies Has a “why” functionality built-in Does not have a “why” functionality built-in as alternative use npm list -g
License checker Can restrict licenses of installed modules and means for supplying license information Running yarn licenses list command will list, in alphabetical order all of the packages that were installed by yarn or yarn install, and give you the license (and URL to the source code) associated with each package. Running npm install license-checker, npx license-checker This will give you a printout of all the licensing details of packages used in your project. Other cool features of the project: Print a summary of licenses used by npx license-checker --summary. This can also be Include in your CI/CD pipeline by providing it with a whitelist or a blacklist of licenses
Fetching packages Yarn stores dependencies locally and fetches them from the disk, instead of sending an HTTP request npm fetches dependencies from the npm registry. however, you can always use npx to fetch dependencies locally if they exist else from the npm registry, which also reduces HTTP call.
@awentzel awentzel added the area:dev-ops Pertains to build, CI, and other dev-ops work label May 19, 2021
@awentzel awentzel added this to the Release 2021-06 milestone May 19, 2021
@awentzel awentzel added this to Triage in DevOps via automation May 19, 2021
@awentzel awentzel changed the title evaluate removing Lerna in favor of Yarn 2 or NPM 7+ audit removing Lerna in favor of Yarn 2 or NPM 7+ May 19, 2021
@awentzel awentzel changed the title audit removing Lerna in favor of Yarn 2 or NPM 7+ build tools: audit removing Lerna in favor of Yarn 2 or NPM 7+ May 19, 2021
@KingOfTac
Copy link
Collaborator

From my own grueling experience over the last three weeks, yarn 2 has some issues with running scripts across all packages in a monorepo, and lerna still seems better suited for running scripts. I am using yarn 2 workspaces with pnp and their no install methodology and it works great. It is just running scripts that has been my biggest issue. I'm sure it is something that can be worked out with the right package config, I just haven't found it yet.

@awentzel awentzel moved this from Triage to To Do in DevOps May 20, 2021
@awentzel
Copy link
Collaborator Author

awentzel commented Jun 9, 2021

Thanks @KingOfTac for your insight.

@Ibrahimmaga Ibrahimmaga moved this from To Do to In Progress in DevOps Jul 13, 2021
@Ibrahimmaga Ibrahimmaga moved this from In Progress to In Review in DevOps Jul 14, 2021
@microsoft microsoft deleted a comment from Ibrahimmaga Jul 14, 2021
DevOps automation moved this from In Review to Done Jul 20, 2021
@Ibrahimmaga Ibrahimmaga reopened this Jul 20, 2021
DevOps automation moved this from Done to Backlog Jul 20, 2021
@Ibrahimmaga Ibrahimmaga moved this from Backlog to In Progress in DevOps Jul 20, 2021
@Ibrahimmaga Ibrahimmaga moved this from In Progress to In Review in DevOps Jul 20, 2021
@awentzel
Copy link
Collaborator Author

@Ibrahimmaga thanks for the detailed analysis, could you please expand on the following for each option (NPM, Lerna, Node, Yarn)

  • Why NPM 7 may not be the best fit for a large enterprise project? Could you also describe performance implications between Node 14 vs 16 as far as this project goes?
  • What is the impact on Code Coverage?
  • What is the impact on Beachball used for publishing?
  • What is the impact on performance?

Thanks,

@bennypowers
Copy link
Contributor

I just got finished moving from yarn1 to npm7 there were some hiccups but thankfully it's all looking good now. I even got to drop wsrun since npm has npm run --if-present built in.

npm7 + changesets is looking pretty good if you can pull it off

@KingOfTac
Copy link
Collaborator

I'm still having issues with typescript not finding package imports when using yarn2 with pnp. I haven't tried npm7 yet to see if it is any better.

@Ibrahimmaga
Copy link
Contributor

@Ibrahimmaga thanks for the detailed analysis, could you please expand on the following for each option (NPM, Lerna, Node, Yarn)

  • Why NPM 7 may not be the best fit for a large enterprise project? Could you also describe performance implications between Node 14 vs 16 as far as this project goes?
  • What is the impact on Code Coverage?
  • What is the impact on Beachball used for publishing?
  • What is the impact on performance?

Thanks,

@awentzel

  1. I was saying is all version of NPM before NPM 7 was not the best choice for large monorepo project because they did not have workspace and package.lock.json file format for rebuilding your package tree from scratch. That's why people were using yarn.
  2. Looking back on node releases, they always have improved in performance. However node version 16 will ship with the v9.0 engine, and still be written in C++ with new features, and will also remove experimental API. This could actually be a win using node version 16 for the creator project since customers do not have to download a library. We should consult with @janechu if customers will be impacted.
  3. code coverage, no impact.
  4. beachball, no impact

@nicholasrice
Copy link
Contributor

@awentzel thanks for summarizing these. Are there any notable criticisms out in the wild of npm 7? It looks like it addresses the major issues that I've encountered with npm.

Tossing node_modules out the window in Yarn 2 makes me a little nervous, just because node_module traversal is literally how packages are resolved by NodeJS. It seems to run counter to the existing ecosystem.

@awentzel
Copy link
Collaborator Author

@Ibrahimmaga how would these commands change and how does NPM 7 process the equivalent logic from our publishing pipeline here?

@Ibrahimmaga
Copy link
Contributor

Ibrahimmaga commented Jul 26, 2021 via email

@janechu
Copy link
Collaborator

janechu commented Jul 26, 2021

I think for the new tooling repository and the creator repository these can both be npm 7 with custom build scripts for anything extra. I would like to see if we can skirt around using Yarn 2 and Lerna if possible, and it seems like this is do-able with the current npm 7 feature set.

@chrisdholt
Copy link
Member

chrisdholt commented Jul 27, 2021

I think I missed it - but is there a reason Lerna is still required with Yarn 2? Yarn includes workspaces as well and I’m not sure we have an accurate comparison if we’re keeping Lerna in the picture when it’s unnecessary from my understanding. Cursory read of above, so perhaps I missed it :)

editing to include a link: https://yarnpkg.com/features/workspaces

@Ibrahimmaga
Copy link
Contributor

I think I missed it - but is there a reason Lerna is still required with Yarn 2? Yarn includes workspaces as well and I’m not sure we have an accurate comparison if we’re keeping Lerna in the picture when it’s unnecessary from my understanding. Cursory read of above, so perhaps I missed it :)

Lerna is not required with Yarn 2. but for a large project, it is nice to have the bootstrapping of the build and get a visual representation that Lerna provides. Otherwise, Yarn 2 and NPM 7 can handle workspaces and crosses dependencies. So I'm sure we don't need Lerna the creator project.

@chrisdholt
Copy link
Member

I think I missed it - but is there a reason Lerna is still required with Yarn 2? Yarn includes workspaces as well and I’m not sure we have an accurate comparison if we’re keeping Lerna in the picture when it’s unnecessary from my understanding. Cursory read of above, so perhaps I missed it :)

Lerna is not required with Yarn 2. but for a large project, it is nice to have the bootstrapping of the build and get a visual representation that Lerna provides. Otherwise, Yarn 2 and NPM 7 can handle workspaces and crosses dependencies. So I'm sure we don't need Lerna the creator project.

Thanks @Ibrahimmaga - I definitely need to read up a bit more on the above. I’m not speaking in terms of creator, but this entire monorepository. I see a lot of references to Yarn 2 + Lerna but I think that’s presenting a bit of a slant towards NPM, mostly due to the fact that the intent here is trying to move away from Lerna so that’s not enticing - but as you’ve said it’s not accurate.

Curious if we have info comparing workspaces in both and if we’ve dug into the node_modules “replacement” text - I just read through after seeing @nicholasrice’s concerns and can appreciate some of their thinking.

I’m not committed to one or the other, but I do want to make sure that we’ve done our due diligence on what matters to a project of our size. Converting is costly and making the right decision here can save headache and headache for all of us.

Thanks for digging into this for us and sharing the findings!

@Ibrahimmaga
Copy link
Contributor

Out of curiosity, does any of the alternatives considered provide the ability to only test changed packages? In our team we do it using lerna ls --since (example). It can be also achieved with Rush but that didn't work well for our use case.

Testing only changed packages could be a good idea for performance. Sometimes the changes affect another package and can go without notice. I think it is better to trade in performance than fixing breaking changes that could take hours to fix. The ideal solution is to test the whole application if there is any change.

@janechu
Copy link
Collaborator

janechu commented Jul 28, 2021

I think it's a good point though, there are several tools being used currently that look up the mono repositories package dependency tree, including beachball. If you can know what the dependents are on a changed package you can only run the tests for the affected packages. I'm going to make a note of this for the migration work for fast-tooling and see if we can't find a good solution because the performance boost would be nice.

@web-padawan
Copy link

Sometimes the changes affect another package and can go without notice.

If this happens, it probably means there are bigger problems related to implicit dependencies not listed in package.json.
Some tools like Rush enforce explicit dependencies and prevent problems with phantom dependencies.

@Ibrahimmaga
Copy link
Contributor

Ibrahimmaga commented Jul 28, 2021

Sometimes the changes affect another package and can go without notice.

If this happens, it probably means there are bigger problems related to implicit dependencies not listed in package.json.
Some tools like Rush enforce explicit dependencies and prevent problems with phantom dependencies.

By using a workspace option, we can use npm command that will run a specific test script within the packages. npm run test --workspace=workspace1 this will go to ./packages/worspace1/package.json file. This will be very useful to speed up local development.

@awentzel
Copy link
Collaborator Author

@Ibrahimmaga please spend some time addressing Chris's concerns here clearly articulating how each handles these scenarios.

  • Improved performance by reducing HTTP calls to fetch packages (per @nicholasrice we should be sure we understand the intent and what's happening here :) )
  • Yarn 2 has a mechanism to restrict certain licenses for packages which can help us ensure OSS compliance, etc...
  • Yarn seems to have superior handling of mismatched packages and resolving them to a single version, something which is quite important with regard to mono-repositories.
  • Yarn includes "why" which can provide insight into why a dependency is included, this could be quite helpful in tracking down security issues, where a package is coming from, etc...
  • Yarn's lock file is far easier to manage conflicts with and understand with regard to dependency resolution.

Thanks @chrisdholt for summarizing your concerns. This is great feedback and Ibrahim will continue to investigate. I think up to this point we were leaning on NPM7 only. Yarn 2 seems to be unstable still for some teams. I think we're all in agreement that a key requirement is reducing dependencies as much as possible. Perhaps Ibrahim can do more research here as well for Enterprise projects. We should be focused on comparing similar projects (apples to apples) and researching to this capacity.

@awentzel
Copy link
Collaborator Author

We will follow this template in term of configuration. The rest will be npm 7 commands. The publishing will be beachball still.

What template @Ibrahimmaga ?

@Ibrahimmaga
Copy link
Contributor

We will follow this template in term of configuration. The rest will be npm 7 commands. The publishing will be beachball still.

What template @Ibrahimmaga ?

Our current pipeline template: (https://github.com/microsoft/fast/actions/runs/1066482693/workflow) someone has asked question if this will changed.

@Ibrahimmaga
Copy link
Contributor

Ibrahimmaga commented Aug 2, 2021

@chrisdholt we are going to be moving tooling to the new fast-tooling repository soon (~3 weeks) which will have largely the same functionality we want from the main FAST repository (multiple packages, publishing, etc.).
I would suggest that we do the work in the fast-tooling repository to have it on npm 7 and find the pain points from actually making the switch. Then we can take another look at the main FAST repository and determine from those lessons how to update it.

Sure, my larger point here is not to block on NPM7 for tooling, but that it seems like a solution has already been resolved upon and I see some conflicting information with regards to requirements and why we're choosing one over the other. Primarily, I would love to see a comparison of features that directly relate to this monorepo and our removal of lerna. How do workspaces compare one vs. the other rather than lumping Lerna in with yarn. The above details that NPM7 has feature parity with Yarn, which I don't think it actually does - there are a few things which Yarn does that NPM does not (likely the same w/ NPM7 doing certain things as well) - do those matter? Perhaps, perhaps not.

TLDR; I'm not sure that I see the above recommendation as clearly as others on the thread do yet - part of that is the confusion I'm seeing by the evaluation of Yarn 2 being tied to an unnecessary addition of Lerna, and the other is that I do think there are some things that Yarn has as advantages which aren't enumerated above:

  • Improved performance by reducing HTTP calls to fetch packages (per @nicholasrice we should be sure we understand the intent and what's happening here :) )
  • Yarn 2 has a mechanism to restrict certain licenses for packages which can help us ensure OSS compliance, etc...
  • Yarn seems to have superior handling of mismatched packages and resolving them to a single version, something which is quite important with regard to mono-repositories.
  • Yarn includes "why" which can provide insight into why a dependency is included, this could be quite helpful in tracking down security issues, where a package is coming from, etc...
  • Yarn's lock file is far easier to manage conflicts with and understand with regard to dependency resolution.

Again, I'm less advocating for Yarn, and more curious if we've done a true comparison given the things that have been problematic or which we'd like to improve. It would be ideal if there were demo repos of each of these we could bootstrap and test against to understand the real-world implications of some of these - though I know that's quite a tall order.

Here a Yarn and NPM comparison side by side by TechGenyz

@rajsite
Copy link

rajsite commented Aug 6, 2021

From the peanut gallery, we have been developing a design system monorepo using fast-foundation and fast-element. Our packages are roughly: components, tokens via amazon style dictionary + Adobe DSP, angular wrappers, and sample apps for vanilla applications and angular applications.

So far we have not had any issues with npm 7 workspaces. The only speed bump so far is the lack of microsoft beachball support for managing the npm package-lock.json file: microsoft/beachball#525

@Ibrahimmaga
Copy link
Contributor

From the peanut gallery, we have been developing a design system monorepo using fast-foundation and fast-element. Our packages are roughly: components, tokens via amazon style dictionary + Adobe DSP, angular wrappers, and sample apps for valilla applications and angular applications.

So far we have not had any issues with npm 7 workspaces. The only speed bump so far is the lack of microsoft beachball support for managing the npm package-lock.json file: microsoft/beachball#525

Thanks for your response @rajsite. I used beachball with yarn, there maybe a new configuration for beachball since the release of NPM 7. You should contact kchau@microsoft.com or @ecraig12345 Elizabeth Creg for beachball support.

@ecraig12345
Copy link
Member

Due to time constraints I mostly only work on beachball as features are needed for my team, unfortunately (same with @kenotron for the most part). But I can help with PR review if someone else wants to implement this.

@Ibrahimmaga Ibrahimmaga changed the title build tools: audit removing Lerna in favor of Yarn 2 or NPM 7+ build tools: Yarn 2, yarn 3 or NPM 7+ for Monorepo Solution Aug 9, 2021
@Ibrahimmaga Ibrahimmaga changed the title build tools: Yarn 2, yarn 3 or NPM 7+ for Monorepo Solution build tools: Yarn 2, yarn 3 and NPM 7+ for Monorepo Solution Aug 9, 2021
@Ibrahimmaga
Copy link
Contributor

@awentzel thanks for summarizing these. Are there any notable criticisms out in the wild of npm 7? It looks like it addresses the major issues that I've encountered with npm.

Tossing node_modules out the window in Yarn 2 makes me a little nervous, just because node_module traversal is literally how packages are resolved by NodeJS. It seems to run counter to the existing ecosystem.

I'm done with this research, but I want to reply to this question. when using Yarn, Npm has to be installed first, Some may claim, NPM and Yarn are 2 differents mono repo solutions, but I see Yarn as an extension for NPM since it is not a standalone application. In the background, Yarn is using NPM to manage node module and have added functionalities that NPM failed short. Without NPM installed, you won't be able to use Yarn.

@merceyz
Copy link

merceyz commented Aug 30, 2021

Yarn is using NPM to manage node module
#4737 (comment)

@Ibrahimmaga Could you elaborate on that statement? Yarn is literally a single JS file so it doesn't need nor use npm to function, it's true we recommend installing it though npm install yarn -g but you don't have to. Now that nodejs/node#39608 has landed, that install step will go away.

Yarn 2 seems to be unstable still for some teams
#4737 (comment)

@awentzel Could you open some issues for the "unstableness" you've encountered? It's difficult to improve if you don't know what you need to improve

But with node_modules/ support added back in
#4737 (comment)

All stable versions of Yarn has node_modules support, so added back in is a bit misleading.

From my own grueling experience over the last three weeks, yarn 2 has some issues with running scripts across all packages in a monorepo
#4737 (comment)

@KingOfTac Could you open an issue for these problems you ran into? https://github.com/yarnpkg/berry

@web-padawan
Copy link

There are two feature requests in Yarn that could enable its usage as a Lerna replacement for our team:

  1. [Feature] yarn workspaces foreach --since yarnpkg/berry#2374
  2. [Feature] version plugin: add --all flag to version command yarnpkg/berry#1229

Both issues have comments with workarounds but IMO having built-in support is a must in terms of stability.

@arcanis
Copy link

arcanis commented Sep 17, 2021

Both issues have comments with workarounds but IMO having built-in support is a must in terms of stability.

I think the bigger picture is that Yarn (2+) lets you easily implement whatever behaviour you need without waiting for the package manager to agree with whatever direction you want to follow, while still benefitting from our core APIs. Here it's changed workspace, but it can also be release workflow, package validation, etc.

@awentzel
Copy link
Collaborator Author

We are continuing to evaluate this. We are trying out a few different scenarios to understand all implications and based on the community feedback and our own conclusions we'll be doing our best to move in the most efficient and productive direction. Thanks, everyone for your feedback on this issue.

@benpsnyder
Copy link

Very nice thread here. I just got done (mostly) implementing Yarn v3 for Nrwl Nx -- see feat(core): implement Yarn v3

Also wanted to share that using @yarnpkg/plugin-patch was insanely awesome and helpful in debugging of issues. Worth looking into!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-ops Pertains to build, CI, and other dev-ops work
Projects
DevOps
  
Done
Development

No branches or pull requests