feat: allow parallel worker builds when worker plugins is a function

[jamsinclair]

Description

Fixes #13367

Background

Currently when there's more than 2 nested workers the Rollup build gets into a deadlock and cannot compile. This was a regression of when worker builds were made sequential (#11218). The reason for sequential building was to solve issues with plugin state pollution in parallel builds

This PR is an attempt at implementing @patak-dev's suggestion in #13394 (comment). By allowing users to provide plugins as a function we can ensure that each worker build gets a separate instance of plugins and hopefully minimise any state pollution problems.

Suggested Solution

Recommend defining worker.plugins in config as a function. When a function is used we run worker builds in parallel because each build can instantiate a fresh instance of the plugins to avoid pollution.

import myWorkerPlugin from 'my-worker-plugin';

export default {
   worker: {
      plugins: () => [myWorkerPlugin()]
   }
}

This new worker config API is backwards compatible and existing configs that have custom worker plugins as arrays will work the same as before.

Let me know if I've misunderstood the solution or whether there's more to be done. Given the complexity I understand if you want to close this PR until your own core team can work on a robust solution.

As always, thanks for all your hard work on this amazing open source tool! 🙇

Additional context

I am unable to get the integration test suite running locally. I followed the contributing guidelines, but no luck. No discernible errors thrown will have to dive into the vitest setup scripts to debug. This is on a Mac M1 Pro, Ventura with Node 18 and 20. Had to debug tests via CI.

---

What is the purpose of this pull request?

• Bug fix
• New Feature
• Documentation update

Before submitting the PR, please make sure you do the following

• Read the Contributing Guidelines.
• Read the Pull Request Guidelines and follow the PR Title Convention.
• Check that there isn't already a PR that solves the problem the same way to avoid creating a duplicate.
• Provide a description in this PR that addresses what the PR is solving, or reference the issue that it solves (e.g. fixes #123).
• Ideally, include relevant tests that fail without this PR but pass with it.

[stackblitz]

Run & review this pull request in StackBlitz Codeflow.

[jamsinclair]

Might need to add some comments. The assumption is that the user worker.plugins function would be side effect free. This allows us to pre-process relevant hook/config updates but still allow fresh plugin instantiation per worker build.

Does that make sense? 😅

[jamsinclair]

I've added two new tests for the worker integration test suite

• Plugin state pollution

  • Parallel worker builds will have a new instance of the state plugin every time and so the count increment will only happen once per build. Highlighting a simple level of isolation when building in parallel
  • Serial worker builds will have the same instance of the state plugin every time and so the count will increment every time a worker includes the referenced file. Highlighting some potential for state pollution or undesired behaviour.

• Deeply nested workers: This asserts that deeply nested workers work in dev and during builds. This is only included in the parallel test file. It will not work in the serial test file as Vite will not compile due to the deadlock bug.

Let me know if you have alternative ideas of how we could simply and effectively test these 🙇

[schickling]

Thanks so much for your work on this @jamsinclair. This addresses a huge issue for my app which currently builds in ~1/50 retries in CI.

I'd highly appreciate any kind of workarounds until this is merged.

[userquin]

I have the SB repro in the linked issue working on my local changing the key for workerConfigSemaphore (it seems the config change and there is a pending promise):

async function bundleWorkerEntry(config, id, query) {
    const processing = workerConfigSemaphore.get({ key: 'vite:worker' });
    if (processing) {
        await processing;
        return bundleWorkerEntry(config, id, query);
    }
    const promise = serialBundleWorkerEntry(config, id, query);
    workerConfigSemaphore.set({ key: 'vite:worker' }, promise);
    promise.then(() => workerConfigSemaphore.delete({ key: 'vite:worker' }));
    return promise;
}

[patak-dev]

Closing this PR as we merged #14685, thanks for your work here @jamsinclair! We added you as a co-author in the PR.