New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: support to run multi script commands specified with regex selector #5871
Conversation
This has still be a POC implementation, so I want your review about CLI interface (current test implementation: https://github.com/pnpm/pnpm/pull/5871/files#diff-377d96dc16ff022df679d40d7013036cc4489527234eabeebf2e91e8afa86eea) and my worries 🙏 |
no
no
I don't understand why would you need separate concurrency setting for this. Just use the already existing one.
I think it is OK to keep the way it is logged now.
I don't think other separators should be supported.
I don't think it is needed. |
}, | ||
}) | ||
|
||
await execa('pnpm', ['add', 'json-append@1']) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why do you need to use json-append for these tests? json-append is useful if you want to check the sequence of the scripts. You can append items to the same file. But you use different files, so you can just create empty files.
like
'build:c': 'node -e "require('fs').writeFileSync('a', '', 'utf8')"',
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Certainly, I used json-append just because other unit tests use this. b92e3bb 👍
@@ -53,6 +54,7 @@ export function rcOptionsTypes () { | |||
'npm-path', | |||
'use-node-version', | |||
], allTypes), | |||
'max-parallel': Number, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is it needed if there is already a workspace-concurrency
setting?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://pnpm.io/cli/recursive#--workspace-concurrency
Set the maximum number of tasks to run simultaneously. For unlimited concurrency use Infinity.
Certainly, s reading this description, —max-parallel
is not needed. I fixed in f256571 👍 I misunderstood —workspace-concurrency
option is just for recursive run
and run
with —filter
option command.
By the above commit, pnpm runs selected scripts parallel with the number of limit specified with workspace-concurrency
. In running scripts specified with both of a wildcard selector and —filter
workspace selector, pnpm flattens specified script commands and runs these from top to bottom.
eg)
In the below package,
a
workspace:build:1
scriptbuild:2
scriptbuild:3
script
b
workspace:build:1
scriptbuild:2
scriptbuild:3
script
and run pnpm --filter '*' run 'build:*'
, pnpm flattens commands to execute into [ a
workspace's build:1
script, a
workspace's build:2
script, a
workspace's build3
script, b
workspace's build:1
script, b
workspace's build:2
script, b
workspace's build:3
script ] and execute these from top to bottom.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Additionally, I make pnpm run
command without recursive execute specified scripts parallel with the number of limit 4
as a default behavior, because pnpm run
command with recursive execute scripts parallel with the number of limit 4
as a default behavior and these should be the same.
So pnpm run
with a wildcard script selector executes specified script parallel as a default behavior, so I implemented --sequential
shorthand option. This shorthand option will specify --workspace-concurrency=1
42b80b3
to
fdedd17
Compare
Thank you for answering my worries!! OK, I am relieved that there is not much difference from your opinion 😄 |
By the way, what do you think about supporting In my opinion, at least |
…ified with selector
…fied with selector in recursive
…rkspace-concurrency
fdedd17
to
514986f
Compare
If you want that, then maybe a regex should be used instead. |
I did not have a strong reason not to use RegExp, so I fixed to use it 👍 bd0aef2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't see new tests that verify matching multiple different script names.
@@ -159,7 +170,13 @@ export async function handler ( | |||
: undefined | |||
return printProjectCommands(manifest, rootManifest ?? undefined) | |||
} | |||
if (scriptName !== 'start' && !manifest.scripts?.[scriptName]) { | |||
|
|||
const multiScriptSelectorSpecified = scriptName.slice(-2) === ':*' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand this. You switched to a regex but still use :*
? How so?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
For example, a regular expression build
without wildcard selector will match build:css
and this is against intention. Therefore, I left this boolean to determine if it is wildcard mode or not.
However I rethought about it and this behavior is against intention as above description and if we want to use RegExp for script matching, a detailed condition judgement process is needed and I thought it is clearer and more maintainable to match script name with simple string prefix matching ( startsWith
) instaed of matching with RegExp so I revert the commit to switch to regex in 7608448.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After all, multiScriptSelectorSpecified
is still needed whether RegExp is used or not. (In the current implementation, RegExp is not used after all.)
This reverts commit bd0aef2.
Hmm…I added some unit tests for verifying matching multiple different script names in 776dd24. Should I also add e2e tests? (which I did not implement from a cost perspective) |
But you wanted to match different script names to run both |
39f3152
to
c02be84
Compare
@@ -164,3 +161,18 @@ export async function runRecursive ( | |||
|
|||
throwOnCommandFail('pnpm recursive run', result) | |||
} | |||
|
|||
function getSpecifiedScripts (scripts: PackageScripts, scriptName: string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why is this function not the same as in run.ts
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In my understanding, pnpm
executes node server.js
in pnpm run start
even if start
script is not defined in package.json. I can find the logic to do so in run.ts
but I cannot in runRecursive.ts
. This diff is occurred from this diff
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we could add a function parameter for controlling this behavior?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No parameter is needed. It is better to write another wrapper for run.st
, which will include the logic for start.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It is also a solution.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I fixed the code to wrap runRecursive
getSpecifiedScripts in `run.tsP
.changeset/silly-flies-care.md
Outdated
"@pnpm/plugin-commands-script-runners": minor | ||
--- | ||
|
||
feat: support wildcard selector to specify multiple scripts to execute" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add more details. This text is going to be on the release page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
4802e46 👍
@@ -0,0 +1,5 @@ | |||
--- | |||
"@pnpm/plugin-commands-script-runners": minor |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
also add pnpm: minor
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
4802e46 👍
@@ -43,13 +45,21 @@ export const RESUME_FROM_OPTION_HELP = { | |||
name: '--resume-from', | |||
} | |||
|
|||
export const SEQUENTIAL_OPTION_HELP = { | |||
description: 'Run specified scripts one after one.', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
description: 'Run specified scripts one after one.', | |
description: 'Run specified scripts one by one', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 Thanks!
d2b73fc
to
4802e46
Compare
@@ -164,3 +161,18 @@ export async function runRecursive ( | |||
|
|||
throwOnCommandFail('pnpm recursive run', result) | |||
} | |||
|
|||
function getSpecifiedScripts (scripts: PackageScripts, scriptName: string) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps we could add a function parameter for controlling this behavior?
const scriptSelector = tryBuildRegExpFromCommand(scriptName) | ||
|
||
// if scriptName which a user passes is RegExp (like /build:.*/), multiple scripts to execute will be selected with RegExp | ||
if (scriptSelector) { | ||
const scriptKeys = Object.keys(scripts) | ||
return scriptKeys.filter(script => script.match(scriptSelector)) | ||
} | ||
|
||
// if scripts in package.json has script which is equal to scriptName a user passes, return it. | ||
if (scripts[scriptName]) { | ||
return [scriptName] | ||
} | ||
|
||
// if a user passes start command as scriptName, `node server.js` will be executed as a fallback, so return start command even if start command is not defined in package.json | ||
if (scriptName === 'start') { | ||
return [scriptName] | ||
} | ||
|
||
return [] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should do the complicated stuff last because in the vast majority of cases it is scripts[scriptName]
.
const scriptSelector = tryBuildRegExpFromCommand(scriptName) | |
// if scriptName which a user passes is RegExp (like /build:.*/), multiple scripts to execute will be selected with RegExp | |
if (scriptSelector) { | |
const scriptKeys = Object.keys(scripts) | |
return scriptKeys.filter(script => script.match(scriptSelector)) | |
} | |
// if scripts in package.json has script which is equal to scriptName a user passes, return it. | |
if (scripts[scriptName]) { | |
return [scriptName] | |
} | |
// if a user passes start command as scriptName, `node server.js` will be executed as a fallback, so return start command even if start command is not defined in package.json | |
if (scriptName === 'start') { | |
return [scriptName] | |
} | |
return [] | |
// if scripts in package.json has script which is equal to scriptName a user passes, return it. | |
if (scripts[scriptName]) { | |
return [scriptName] | |
} | |
// if a user passes start command as scriptName, `node server.js` will be executed as a fallback, so return start command even if start command is not defined in package.json | |
if (scriptName === 'start') { | |
return [scriptName] | |
} | |
// if scriptName which a user passes is RegExp (like /build:.*/), multiple scripts to execute will be selected with RegExp | |
const scriptSelector = tryBuildRegExpFromCommand(scriptName) | |
if (scriptSelector) { | |
return Object.keys(scripts).filter(script => script.match(scriptSelector)) | |
} | |
return [] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Indeed. acf55fd
if (command.length < 3) { | ||
return null | ||
} | ||
if (command[0] !== '/' || command.lastIndexOf('/') < 1) { | ||
return null | ||
} | ||
try { | ||
return new RegExp(command.slice(0, command.lastIndexOf('/')).slice(1)) | ||
} catch { | ||
return null | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Support /x/imgy
.
if (command.length < 3) { | |
return null | |
} | |
if (command[0] !== '/' || command.lastIndexOf('/') < 1) { | |
return null | |
} | |
try { | |
return new RegExp(command.slice(0, command.lastIndexOf('/')).slice(1)) | |
} catch { | |
return null | |
} | |
// https://github.com/stdlib-js/regexp-regexp/blob/6428051ac9ef7c9d03468b19bdbb1dc6fc2a5509/lib/regexp.js | |
const isRegExpStr = /^\/((?:\\\/|[^\/])+)\/([imgy]*)$/.test(command) | |
if (!isRegExpStr) { | |
return null | |
} | |
// https://stackoverflow.com/a/874742/6596777 | |
const match = command.match(new RegExp('^/(.*?)/([imgy]*)$')) | |
if (!match) { | |
return null | |
} | |
try { | |
return new RegExp(match[1], match[2]) | |
} catch { | |
return null | |
} |
Ref:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There are other flac chalactor to present flag https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/RegExp/flags other than imy
, and these are defined in EcmaScript RegExp specification. I think it is hard to keep up with RegExp flag for ecmascript starndart so it is better to simply split wih /
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! I have modified the code:
// https://github.com/stdlib-js/regexp-regexp/blob/6428051ac9ef7c9d03468b19bdbb1dc6fc2a5509/lib/regexp.js
const isRegExpStr = /^\/((?:\\\/|[^\/])+)\/([dgimuys]*)$/.test(command)
if (!isRegExpStr) {
return null
}
// https://stackoverflow.com/a/874742/6596777
const match = command.match(new RegExp('^/(.*?)/([dgimuys]*)$'))
if (!match) {
return null
}
try {
return new RegExp(match[1], match[2])
} catch {
return null
}
Result:
console.log(/build:.*/dgimuys.test("build:web")) // true
Excess flags do not affect the results.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In what case would regex flags be useful? A regex is already more powerful than it needs to be for this simple task of selecting scripts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, only the i
flags are useful. But if flags are not supported, I think it's fine.
We need to discuss what will happen if the user types pnpm run /build:*/ig
:
- Ingore
flags
and printwarning
- Ignore
flags
- Report an error
- Consider this as
scriptName
- Consider this as Regex flags.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- If ignore
flags
:
// https://github.com/stdlib-js/regexp-regexp/blob/6428051ac9ef7c9d03468b19bdbb1dc6fc2a5509/lib/regexp.js
const reRegExp = new RegExp(/^\/((?:\\\/|[^\/])+)\/([dgimuys]*)$/)
const match = command.match(reRegExp)
if (!match) {
return null
}
// if need print warning
if (match[2]) {
logger.warn("not support flags and will ignore")
}
try {
return new RegExp(match[1])
} catch {
return null
}
- If support
flags
:
// https://github.com/stdlib-js/regexp-regexp/blob/6428051ac9ef7c9d03468b19bdbb1dc6fc2a5509/lib/regexp.js
const reRegExp = new RegExp(/^\/((?:\\\/|[^\/])+)\/([dgimuys]*)$/)
const match = command.match(reRegExp)
if (!match) {
return null
}
try {
return new RegExp(match[1], match[2])
} catch {
return null
}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd vote for either reporting an error or consider it as script name. Although the second one may be confusing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would like to vote for reporting an error or ignoring flag and warning it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'd vote for ignoring the flags and warning it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I updated the tryBuildRegExpFromCommand
to report the error if RegExp flag is specified in the passed script command 67f62a1 👍
Some tests are failing. |
function getSpecifiedScripts (scripts: PackageScripts, scriptName: string) { | ||
const specifiedSelector = getSpecifiedScripts(scripts, scriptName) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry 🙏 I fixed the code to fix CI error
0049863
to
c93730e
Compare
c93730e
to
0d81ca4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit
const scriptSelector = tryBuildRegExpFromCommand(scriptName) | ||
|
||
// if scripts in package.json has script which is equal to scriptName a user passes, return it. | ||
if (scripts[scriptName]) { | ||
return [scriptName] | ||
} | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
First scripts[scriptName]
then tryBuildRegExpFromCommand
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good Catch! Thanks! 46522b3
} catch (_err: any) { // eslint-disable-line | ||
err = _err | ||
} | ||
expect(err.message).toBe('RegExp flag is not supported in script command selector') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
1485f79 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
Ticket
Fixes: #3563
Details
support function to run multiple script commands specified with RegExp selector like
/build:.*/
.If you run
pnpm run /build:.*/
, matchedbuild:css
,build:html
,build:js
... will be fired, and unmatchedlint:css
,start
will not be fired. You can also specify--max-parallel
option to specify number of limits to run scripts specified with a wildcard script selector parallel.--max-parallel
default value is 1 (= sequential). If you specify--max-parallel 2
, pnpm runs matched script commands parallel 2 each.RegExp flags like
/~~/ig
are not supported because flags are not useful.Tests
in this repository.
run
cd b && pd run build:*
run
cd b && pd run --max-parallel 3 build:*
run
pd --filter * build:*
My worries points
As the above description, I considered supporting selectors like
XXX:*:YYYY
in the second example, but I felt this will cause an unnecessary complexity for a small number of use cases, so I only support selectors ending with:*
in the current implementation, but should we support this feature?run
,run-script
command?In the current implementation, the wildcard script selector is supported in
run
andrun-script
command, but It seems also good to create a newrun-all
command for this wildcard script selector, and not support the wildcard script selector inrun
andrun-script
command. (butrun-all
command may cause unnecessary complexity…)pnpm supports workspace function, and pnpm runs scripts in matched workspaces with
—filter
options parallel. The number of limits of this parallel script execution over workspaces is specified withworkspace-concurrency
. I was wondering about the balance of the number of limits for parallel execution between parallel script execution over workspace and parallel script execution with wildcard script selector.In the current implementation, the number of limits of parallel execution is
${—max-parallel} * ${workspace-concurrency}
, but I felt this is confusing perhaps...In the current implementation, pnpm invokes
runLifecycleHook
module for each the matched scripts with workpsace filtering with—filter
option and a wildcard script selector filtering andrunLifecycleHook
module initializesnpmLog
function instance per call (https://github.com/pnpm/pnpm/blob/main/exec/lifecycle/src/runLifecycleHook.ts#L91), sonpmLog
function outputs logs separately and log grouping per workspace does not work like the below screenshot.In my understanding, pnpm writes log output to
Rx.ObservableContext
and logs it to the command line, structured withcli/default-reporter
.It is necessary to write the log of the execution of the script matched by the wildcard script selector in the same log context per workspace without destroying the existing context in order to support log grouping per workspace. I could not come up with a way to realize it…Do you know how to do it? (Or maybe it's fine as it is now...?)
/
symbol to separate the structure of a wildcard script selector?In the current implementation, only
:
symbol is supported to separate the structure of a wildcard script selector, soXXX:*
works butXXX/*
does not work. I felt that supporting also/
symbol will cause unnecessary complexity, but should I support this?I did not implement script selector with regex because I felt this will cause an unnecessary complexity for a small number of use cases, but should I support this?