[RRFC] Execute workspace commands in good order

[chrisui]

Motivation ("The Why")

We would like to have a very thin layer of script management over npm workspaces to run scripts in packages, using the npm cli as a dependency resolver and symlinker.

To run scripts within workspaces which may have dependencies we would want to be able to enforce a good order which is based on the dependency graph. Right now, as far as I can see, this is left for users to implement however as npm tooling owns the dependency graph and implements the ability to provide multiple --workspace options to commands then it would be a sane default that these would be run in good order.

Example

Given the workspaces below:

/packages
  /a
  /b    # depends on /a
  /c    # depends on /b

The following script:

npm run build -w c -w b -w a

Should guarantee the following run order:

npm run build -w a
npm run build -w b
npm run build -w c

Currently this is not the case and the run order seems dependent on user input order, which means a whole lot of complexity is left on the outside of npm where it is well-defined.

Our Current Script

Note that the output of this script is not guaranteed, or designed, to be in good dependency running order.

#!/usr/bin/env bash

output_prefix=""
base_git_ref="HEAD~1"

# Get the options
while getopts "p:b:" option; do
  case $option in
    p)
      output_prefix=$OPTARG;;
    b)
      base_git_ref=$OPTARG;;
  esac
done

# 0. use git to get changes between current commit and a base
# 1. remove the " 0.5%" stat output prefix
# 2. normalize paths to the workspace level (ie. only two levels deep)
changed_workspace_dirs=$(\
  git diff --dirstat=files,0 HEAD..${base_git_ref} \
  | sed 's/[ .0-9]*% //g' \
  | cut -d/ -f 1-2
)

# nothing to do if no dir changes
if [ -z "$changed_workspace_dirs" ]
then
  exit 0
fi

# get npm to tell us what is depending on the workspaces we've changed
# ignoring any errors thrown from paths it does not recognise (it'll just be a noop)
explained=$(npm explain ${changed_workspace_dirs} --json) || true

# 0. pipe to jq for processing structured json
# 1. recursively get packages by looking for name (see: https://gist.github.com/pedroxs/f0ee8c515eea0dbce2e23eea7c048e10)
# 2. we only care about the unique names
# 3. get the raw names (ie. not json "string")
workspaces=$(\
  echo "$explained" \
  | jq 'map( .. | objects | with_entries(select(.key | contains("name"))) | select(. != {}) )' \
  | jq 'map(."name") | unique' \
  | jq -r '.[]'
)

prefixed_workspaces=$(echo "$workspaces" | sed "s/^/$output_prefix/")

echo "$prefixed_workspaces"

How

Current Behaviour

See Example (NON-deterministic order of commands given knowledge of workspace dependency graph)

Desired Behaviour

See Example (deterministic order of commands given knowledge of workspace dependency graph)

References

• Lerna Package Graph
• Issue discussing lerna run order as a user
• wsrun for Yarn (and actually seems to work ok so far with nom 7)

[lukekarrys]

Thank you for this RRFC and sharing you current script! There is a new RRFC for this (#548) that was discusses in the last RFC call and has gained some traction. I'm going to close this in favor of that in order to consolidate the discussion, but please feel free to comment there if it doesn't fit your use case (I think it does from reading this issue).