RFC: npm workspaces - Working with workspaces

[ruyadorno]

npm Workspaces: Running commands

Summary

Introduces basic support for running npm commands across a project's workspaces.

Motivation

The need for running npm commands across a project's workspaces was surfaced by the community during the discourse spun out of the initial npm workspaces RFC.

Detailed Explanation

Following up on the original npm workspaces RFC are the following command-related additions/changes:

• A subset of the standard npm commands need to be made aware of npm workspaces
• It's desired to have a standard way to filter out workspaces in which to run specific npm commands

Rationale and Alternatives

The ability to run npm commands across a project's workspaces is essential to effecient management of, & enabling, complex developer workflows.

The following are possible alternatives to unlock this functionality:

• Do not implement further support to manage running commands across workspaces, keep the npm workspaces set of features to a minimum
• Defer to other tools from the ecosystem - such as Lerna - to solve the problem of running top-level commands across workspaces
• Implement the proposed features of this RFC as a standalone package apart from the npm cli

Implementation

1. Run commands across all child workspaces

Create a new npm cli config, --workspaces (aliased to --ws), that can route any of the supported subcommands to run in the context of the configured workspaces as long as a workspaces configuration field is properly defined in package.json

There are currently five distinct categories of "workspace-awareness" that existing subcommands will belong to:

1. Commands that just need context from package.json

Commands that, from a user's point of view, are the equivalent of: cd <workspace-name> && npm <cmd>.

• docs
• doctor
• diff
• dist-tag
• pack
• publish
• repo
• set-script
• unpublish
• view

2. Custom implementation

2.1. Reads from installed dependency tree

General class of helper commands that load from the installed tree to produce some form of useful output.

• audit
• explain
• fund
• ls
• outdated

2.2. Modify installed dependency tree

The set of commands that will modify an install tree (from an implementation stand-point, these are just arborist.reify proxies).

• ci
• dedupe|find-dupes
• install-ci-test
• install-test
• install
• link
• rebuild
• update
• uninstall

2.3. Other

Commands that need a special/custom "workspace-aware" implementation outside of the context of reading/writing to the install tree.

• exec
• init
• run-script|restart|start|stop|test
• version

3. Unsupported

This category of npm cli commands is completely unrelated to anything that the current working directory could affect. This includes a number of registry-specific helper/management operations which don't make sense/have no effect within the context of a workspace. Trying to run these commands with a workspace flag/config will exit with a descriptive error code.

• adduser|login
• bin
• birthday
• cache
• completion
• config|get|set
• deprecate
• edit
• explore
• help
• help-search
• hook
• logout
• org
• owner
• ping
• prefix
• profile
• search
• shrinkwrap
• star
• team
• token
• unstar
• whoami
• workspaces

Example:

Running tests across all configured workspaces:

├── package.json { "name": "foo", "workspaces": ["dep-a", "dep-b"] }
├── dep-a
│   └── package.json { "version": "1.0.0" }
└── dep-b
    └── package.json { "version": "1.3.1" }

$ npm test --workspaces

> foo@1.0.0 test /Users/username/foo
> echo "Error: no test specified" && exit 1

Error: no test specified
npm ERR! Test failed.  See above for more details.

> dep-a@1.0.0 test /Users/username/foo/dep-a
> done

> dep-b@1.3.1 test /Users/username/foo/dep-b
> done

2. Filter a subset of workspaces

Filter is done via named argument (--workspace, short: -w) and here are some of the reasons why we decided to go that route:

• Lerna filters were the starting point but early in the discussion some other considerations were brought forward, specially the many pitfalls of supporting globs in the variety of supported shells and operational systems.

3. Examples

Given the results of all this preliminar investigation, the preferred way to run a command in the context of a single workspace is to use the named --workspace argument or its -w short alias, e.g:

In a project with the following structure:

./
├── package.json { "name": "root", "workspaces": ["packages/foo", "packages/bar"] }
└── packages/
    ├── foo/
    │   └── package.json { "name": "foo", "version": "1.0.0" }
    └── bar/
        └── package.json { "name": "foo", "version": "1.0.0" }

You can run tests for the foo workspace from the root of your project, with the following syntax:

npm test --workspace=foo

Install dependency example:

Add tap as a dependency for all of your configured workspaces:

npm install tap --workspaces

Note: Globs are not supported as a valid --workspace argument value, the proposed alternative is to use npm <cmd> --workspace=<dir> in which <dir> is a folder containing multiple workspaces.

Prior Art

Previously

• npm workspaces

Filtering a subset of workspaces

• lerna filter-options
• Yarn v1 workspace cmd
• Yarn v1 workspaces cmd
• Yarn v2 foreach include/exclude
• pnpm Filtering

[ruyadorno]

Action items from OpenRFC call:

• Document backstory on why we want to use positional arguments vs named config in the RFC
• Maybe also document in some form the initial discussion from the OpenRFC call

[mtdev2]

viewed changes

[wesleytodd]

One thing not mentioned in the RFC is what happens when a command is run in a sub-package. The assumption is that it ignores the workspace and just runs in the current package, I confirmed this with @ruyadorno via slack, but it is not explicit in this doc. Maybe it is not necessary to add, but it was context I was missing.

[ruyadorno]

Action Item from OpenRFC call:

• Create a poll to collect feedback on preference for the filter spell
• Expand list of supported top-level commands to make sure it lists all commands (excluding only global commands that do not apply to packages)

[ljharb]

Moving my comment from #160:

• short names should only be aliases for nice and verbose long ones; ie, ws can be short for workspace but the name of the command is "workspace" and they have to do the same thing
• If the semantic is only "provide a single subpackage/workspace" then --workspace= (on any npm or npx command) makes sense to me. however that's way less useful than "provide a glob pattern that matches 0-n workspaces", which would need a different name, like "scope" or "filter" or something.

[ruyadorno]

Notes from the OpenRFC call:

• make sure groups/alias also have an alternative way to be declared as cli args

[ruyadorno]

heads up, workspaces support went out in npm@7.7.0 for the first couple of commands: exec and run-script

[glen-84]

What is the status of npm install <pkg> -w <workspace-name>?

Is the only option to edit the workspace package.json files manually?

What is the ETA for this functionality?

(Unrelated: I find the naming a bit strange, as a workspace usually refers to a set of projects, not to the projects themselves.)

[ruyadorno]

What is the status of npm install <pkg> -w <workspace-name>?

⏳ in progress: npm/arborist#257

Is the only option to edit the workspace package.json files manually?

It's not necessarily the only choice but I'd be tempted to say it's the safest, recommended choice for now.

What is the ETA for this functionality?

It might land in the next couple of weeks, stay tuned for the upcoming releases.

(Unrelated: I find the naming a bit strange, as a workspace usually refers to a set of projects, not to the projects themselves.)

I see your point and it's def a valid concern that I don't think it was mentioned before and in the RFC itself "workspace" is a noun used interchangeably meaning both the feature/project as npm workspaces and referring to nested packages as workspaces. It made me think about a bunch more reasons for picking the name that I'd like to document in the RFC too:

• The workspace config is an option that enables both "running in workspace mode" + "filter by these names/dirs"
• -w is an available shorthand and a much simpler one to relate to "workspaces" in a general sense

[ljharb]

@ruyadorno i've brought it up a number of times; to me a "workspace" is the repo, and the individual chunks in it are "packages" (ie, the default directory name in lerna for these).

[ruyadorno]

Sorry about it @ljharb I know you brought up the terminology confusion multiple times 😬

[zkochan]

I agree with @ljharb and @glen-84 on this. In pnpm we use workspace for calling the repository and packages inside the workspace are called projects or packages.

Some IDEs use the same terminology I believe. And Angular projects also use the workspace terminology for noting sets of packages. I don't know why Yarn decided to use workspace for calling packages in a monorepository (Yarn calls the monorepository a project and the packages inside the monorepository are called workspaces).

Maybe you should make a poll and see what your users would prefer. When I did a poll between pnpm users, for the majority workspaces meant a set of packages or projects, not vice versa.

[ruyadorno]

Thanks for the feedback @zkochan @ljharb @glen-84, while we def appreciate the intention of your suggestion, the consensus within the @npm/cli-team team is to keep the current nomenclature for now - we can def revisit this decision in the future but at this point I'm going to close this PR as the RFC discussed in here has been ratified (and many of the commands have actually already landed in the cli).

I would encourage anyone interested in adjusting that terminology to follow up either in the form of a RRFC issue or as a GitHub Discussion in this repo.

[ruyadorno]

another important update to folks still subscribed to this thread, most of the remaining planned workspace support landed in npm@7.14.0, such as support to add/remove deps to workspaces, e.g: npm install <pkg> -w <workspace-name> 🎉

cc @glen-84