[Bug?]: Executing other scripts within a script causes unavoidable additional start-up

[yinzara]

Self-service

• I'd be willing to implement a fix

Describe the bug

We have a large monorepo with a 4mb .pnp.cjs file.

When any yarn x command is executed, the startup time is between 1.5 and 2.5s (depending on the machine environment) which I assume is because of the large pnp file.

You can test this with a simple package.json with scripts:

"scripts": {
  "one": "echo one"
}

time yarn echo:

real    0m1.458s
user    0m1.704s
sys     0m0.218s

While this might not be a very big deal when you're just executing a simple script that runs a single command, when you start to have scripts that call other scripts, the problem compounds itself.

We often make scripts that execute multiple other scripts within the same package.json like:

"scripts": {
    "build": "yarn check:types && yarn build:js && yarn test",
    "check:types": "tsc ....",
    "build:js": "babel ....",
    "test": "jest"
}

or that call one script from another in succession to build a more complex command like:

"scripts": {
  "do:foo:": "bang bash",
  "do:foo:bar:": "yarn do:foo:",
  "do:foo:bar:bang": "DING=dong BASH=bang yarn do:foobar: -- bingo bango"
}

What we did not realize is for every execution of yarn, we were incurring the cost of the startup time of yarn pnp.

As you can see from the below reproduction steps, the more executions of yarn, the more the startup time multiplies.

I can find no way around this right now.

I can see a few potential solutions to this problem:

1. Fix .pnp.cjs (and yarn) to load very quickly when already loaded in the same process. This would be the best option IMHO :)
2. Fix the script execution environment that when its executing yarn that is has some special handlings to not reload the .pnpjs. I think this would be the most ideal though I'm betting there's some major consequences to doing this that would be a big deal to avoid, but I thought I'd suggest it.
3. Make another specialized way (some kind of special syntax) for executing another yarn script. I wouldn't be opposed to this though it would require code changes for those who use yarn and would make some special syntax that again varies from npm and it would have to be something pretty special that would never intersect with a real command.

Any other ideas? Am I missing something?

To reproduce

You need a pretty big monorepo or a package.json with a lot of dependencies. We have 500~. Something to make your .pnp.cjs pretty large (ours is 4mb)

Start by yarn install to generate your .pnp file then:

"scripts": {
    "one": "echo one",
    "two": "yarn one",
    "three": "yarn two",
    "four": "yarn three",
    "five": "yarn four",
    "six": "yarn five && yarn four && yarn three"
}

time yarn one:

real    0m1.458s
user    0m1.704s
sys     0m0.218s

time yarn five:

real    0m9.033s
user    0m9.584s
sys     0m1.300s

time yarn six:

real    0m22.385s
user    0m24.612s
sys     0m3.339s

Environment

System:
    OS: macOS 11.6.5
    CPU: (20) x64 Intel(R) Xeon(R) W-2150B CPU @ 3.00GHz
  Binaries:
    Node: 14.18.1 - /private/var/folders/hw/120lj3wj5rb1pmb_x_s5tjp40000gn/T/xfs-9b58a563/node
    Yarn: 3.1.1 - /private/var/folders/hw/120lj3wj5rb1pmb_x_s5tjp40000gn/T/xfs-9b58a563/yarn
    npm: 8.12.2 - ~/.nvm/versions/node/v14.18.1/bin/npm

Additional context

No response

[merceyz]

Duplicate of #2575

[merceyz]

In my testing it isn't the loading of .pnp.cjs that causes the major slowdown, it's all the other work the Yarn processes does.

[yinzara]

So while I agree this is partially caused by #2575, I feel like this is still a separate issue that could potentially be fixed separately from #2575.

#2575 is trying to fix how slow it is to start yarn and the likelihood of it being fixed without major changes to yarn are low.

This issue is about running script commands that executes other script commands causing the startup time to be multiplied.

There could potentially be a fix for this issue without fixing #2575 and even if you fix #2575, this issue could still remain since it's caused by a multiplicative effect.

Even if the startup time is really slow, as long as it only loads once we could execute multiple scripts and it wouldn't be that slow.

If we could fix the script execution environment to detect when you're executing the yarn command and instead of actually executing it, do some special handling to execute it logically within the running process.

A specialized custom syntax could also be a potential solution to execute a yarn script from within a script.

This is a pretty common use case that I think merits at least being tracked.