Importing a file from a web worker that imports said web worker causes Vite to build forever #7015
Comments
|
i have nearly the same problem and wondering if anyone has a workaround in mind... |
|
I found a partial workaround to at least prevent the infinite loop and get a build output: in vite.config.ts : build: {
commonjsOptions: {
dynamicRequireTargets: [
'src/MyWebWorker.ts',
],
},
},
...this avoids the loop during the transforming step, but does not transpile the webworker. rather it creates a single line file with require("/Absolute/Pathto/src/MyWebWorker.ts"); |
|
I have a simplified reproduction of this infinite build https://stackblitz.com/edit/vitejs-vite-oqqjnr?file=worker.js,main.js,package.json&terminal=dev |
Thanks for the much simpler reproduction. However, this may be a different bug? Instead of Vite showing I forked your project. using |
I guess they are the same under the hook after this change vite/packages/vite/CHANGELOG.md Line 444 in 77865b4 |
This comment was marked as spam.
This comment was marked as spam.
|
I also ran into this while trying to create a single file typed web worker. https://stackblitz.com/edit/vitejs-vite-9zt7v2?file=src/worker.ts It works in dev but goes into a infinite loop during build. What I was trying to do is build something like https://nodejs.org/api/worker_threads.html, where the script for both the client and the worker can be in the same file. This is desirable in TypeScript in order align the type of data passed into the worker from the client. Before this gets resolved, I now have to split up my worker into multiple files. |
|
I am running into the same issue. This is particularly in a Vue project and my initial solve was to use inversion of control by having the module which was originally calling let ModuleWorkerDefinition;
export function setWorkerDefinition(workerDefinition: () => Worker) {
ModuleWorkerDefinition = workerDefinition;
} So that when something needs to create a worker it calls a method, call it export function useWorker() {
return ModuleWorkerDefinition();
}And lastly the new Worker call is abstracted out into a separate module, call it // some import that causes your circular dependency issue because it in turn somewhere imports `useWorker.ts`
import { setWorkerDefinition } from '@/useWorker';
setWorkerDefinition(() => {
return new Worker(/* worker stuff here */);
})This means that In Vue this was pretty easy. In the |
|
In talking through this with one of my teammates I think it makes sense why this is happening. Can someone confirm that this is how the bundler is working and that this logic makes sense? It seems to us that this is about the way that the bundler should be working:
I understand that there is a lot more to the process here, but as a whiteboard explanation of the process, I think that this is pretty close to what it seems to be doing? If that is the case, then assuming that there is some process/memory isolation in step 3 it would make sense that the circular loop would case the cycle to repeat over and over until you run out of memory. However, it seems like a reasonable solution to essentially create a dictionary where the keys are the import paths and the values are the files that will ultimately be created so that you can effectively reuse the output of the sub-process spawned in step 3 at any import, even if that import is inside of the worker itself, right? Because in the end this is going to show up in the bundled, transpiled, minified code as Is this something that it would even be possible to write a Vite/Rollup plugin for, or is this just too core to the way that those tools work? |
|
I had the same issue, and have found a workaround after attempting for several days. Hope this can help someone. The key idea is forcing the vite bypassing let scriptForWorker = "/path/to/myWorker.js"
const worker = new Worker(scriptForWorker, { type: 'module' })The
import { defineConfig } from 'vite';
export default defineConfig({
build: {
rollupOptions: {
input: {
main: "index.html",
worker: "path/to/worker"
}
},
manifest: true
}
});
let scriptForWorker = "path/to/worker.ts";
if(!import.meta.env.DEV){
// in production, we get the js from manifest.json
// in `npm run dev` mode, using orignal path directly
const response = await fetch("/manifest.json");
if(response.ok){
const manifest = await response.json();
scriptForWorker = manifest[scriptForWorker].file;
}
else{
return Promise.reject(response.statusText);
}
}
// create the Worker
const worker = new Worker(scriptForWorker, { type:'module' });
// or the SharedWorker
const sharedWorker = new SharedWorker(scriptForWorker, { type:'module' });
npm run build
# or
npm run dev |
|
We had the exact same case (worker -> importing file -> importing same worker). However, instead of hanging, we got: |
|
we also get this hang. any update? |
|
From the error logs, it seems that this happens from trying to bundle workers as CommonJS? Would the error go away if worker constructor files were instead just treated as entry points for ESM bundling? I understand that some apps will want backwards compat through opt-in or opt-out, but all modern browsers and runtimes support module workers. |
|
I believe for a while the problem was masked by the nested workers regression (#13367). You can see this in other developer outputs above. The error would look like: That issue was fixed in #14685 and released in v5.0.0-beta.11. The original reported issue is now present for nested workers and re-occuring from v5.0.0-beta.11. After the next major release I think there will be an uptick in people reporting this hanging build issue as circular imports is a common pattern when working with nested workers (pun unintended) and often added automatically by dev tooling. |
|
This issue is still present in v5.0.0-beta.11. As mentioned in #14499 (comment), it can be circumvented by doing: Detailsimport { defineConfig } from 'vite';
// https://github.com/vitejs/vite/blob/ec7ee22cf15bed05a6c55693ecbac27cfd615118/packages/vite/src/node/plugins/workerImportMetaUrl.ts#L127-L128
const workerImportMetaUrlRE =
/\bnew\s+(?:Worker|SharedWorker)\s*\(\s*(new\s+URL\s*\(\s*('[^']+'|"[^"]+"|`[^`]+`)\s*,\s*import\.meta\.url\s*\))/g
// https://vitejs.dev/config/
export default defineConfig({
build: {
target: 'esnext',
},
server: {
host: 'localhost',
headers: {
'Cross-Origin-Embedder-Policy': 'require-corp',
'Cross-Origin-Opener-Policy': 'same-origin',
'Cross-Origin-Resource-Policy': 'cross-origin'
}
},
optimizeDeps: {
esbuildOptions: {
target: 'esnext'
}
},
worker: {
format: 'es',
// https://github.com/vitejs/vite/issues/7015
// https://github.com/vitejs/vite/issues/14499#issuecomment-1740267849
plugins: () => [
{
name: 'Disable nested workers',
enforce: 'pre',
transform(code: string, id: string) {
if (code.includes('new Worker') && code.includes('new URL') && code.includes('import.meta.url')) {
return code.replace(workerImportMetaUrlRE, `((() => { throw new Error('Nested workers are disabled') })()`);
}
}
}
]
}
});Note that for Emscripten-generated code this would mean that the |
|
I'd like to second @lgarron's idea for not separately bundling module workers. Seems like it would simplify many things. No need to worry about nested workers, parallelized builds, circular-ish dependencies and separate worker plugins. Plus reduced code duplication as modules can be shared between main entry points and module workers. Are there any obvious or hidden pitfalls that would make this unappealing? @patak-dev @sapphi-red Some smaller ones that I can think of are:
|
|
I've attempted to create an experimental plugin that would make Vite chunk module workers instead of bundling. It seems to work for my use case and solves the circular dependency issue (Gist of plugin source). There's a few caveats that showed up:
There's likely other things that could break because the plugin simply tricks Vite into thinking the worker is a dynamic import of an ES module. |
|
for me it worked for opencv.js wasm workers with the config from #7015 (comment) , thanks a lot! |
|
Fixing this may solve nuxt/nuxt#22966 |
|
I'm having a similar problem. Simple web workers are fine, but any code that imports classes & instantiates a new object inside a web worker always fails. |
|
While I appreciate a partial fix in #16103 , I'd like to note that that it does far from solving the issue in general. Trying to run #16103 states that:
I appreciate that it might not be possible to fix the current situation super easily, but it's not a matter of popular conventions. Some of us don't have a choice about the recursive dynamic imports in our code, unless we want our code to fail in other bundlers. 😢 Would it be possible to reopen one of these issues to keep track of the general problem?
Thanks, |
I do think it solves most cases. I actually tested with all the repros I found and all of them worked (without throwing errors on build) as I stated in the PR. Your repro does not match the "complex case" condition. By using the main branch (eef9da1) Vite and setting
If it doesn't work on your computer, there's probably something wrong with your self-built Vite. If not, maybe there's a bug that doesn't happen on my computer? |
That's reassuring to hear! I tried it again, and I still can't get it to work — do you have more details on how to test? This is what I'm doing: # In a vite checkout:
pnpm i
pnpm run build
cd packages/vite
npm link
# In a project:
npm link viteRegardless of what I try to put in |
|
Aha! I forgot that the export default {
worker: {
format: "es"
}
}🤩🤩🤩 This is fantastic to see, but unfortunately it does mean people using our library will still get a confusing error out of the box. |
We won't be able to do that for now. ESM workers are not supported by browsers we specify as the default value of |
what about typescript? |
Describe the bug
I have my project set up where I have an index file that exports everything in the folder. This is so I can solve a circular dependency.
When I created a web worker, I had it import from the index file. One of these files in the index directly imported the web worker using
?worker. This worked fine in development. When I went to build, however, Vite seemed to get stuck in an infinite loop and only stopped when it ran out of memory.Reproduction
https://github.com/hf02/vite-infinite-buildMuch simpler reproduction, thanks to @zxch3n: https://stackblitz.com/edit/vitejs-vite-u59lcw?file=worker.jsSystem Info
System: OS: Windows 10 10.0.22000 CPU: (16) x64 AMD Ryzen 7 3700X 8-Core Processor Memory: 9.38 GB / 15.95 GB Binaries: Node: 16.14.0 - ~\AppData\Local\nvs\default\node.EXE npm: 8.3.1 - ~\AppData\Local\nvs\default\npm.CMD Browsers: Edge: Spartan (44.22000.120.0), Chromium (98.0.1108.56) Internet Explorer: 11.0.22000.120Used Package Manager
pnpm
Logs
Validations
The text was updated successfully, but these errors were encountered: