Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to use NodeJS worker_threads module #18540

Closed
d0peCode opened this issue May 31, 2019 · 52 comments
Closed

Unable to use NodeJS worker_threads module #18540

d0peCode opened this issue May 31, 2019 · 52 comments

Comments

@d0peCode
Copy link

Issue Details

  • Electron Version:
    v5.0.2
  • Operating System:
    Windows 10

Expected Behavior

On app ready start nodejs worker which consoles string.

Actual Behavior

App crash immediately after run

To Reproduce

Additional Information

I used electron-forge to quick build app.

@walmat
Copy link

walmat commented Jun 25, 2019

Having the same issue with Electron 4.x.x and Node 12.4. Same behavior and setup. Have you found a fix for this?

I've tried:

  1. export NODE_OPTIONS=--experimental-worker in both the runtime script && in my ~/.bash_profile
  2. Running electron as a node process via the RUN_ELECTRON_AS_NODE=1 flag

I've switched to using a global.window.Worker (web worker) for now, but would love to implement this lib in my project as soon as possible. If anyone can shed some light on this, or share a working example, that'd be great!

edit: @sofianguy maybe you can lead us in the right direction, or give us a yes/no as to whether this is working in v4 or 5

@d0peCode
Copy link
Author

Having the same issue with Electron 4.x.x and Node 12.4. Same behavior and setup.

AFAIK Electron ship its own node fork, so your local 12.4 doesn't really matter.

Have you found a fix for this?

No, still waiting for response. There is pinned discussion #18397 which might be relevant somehow.

@infFenrir
Copy link

Having the same issue with Electron 5.0.6 and Windows 10 :(
App crash...

@d0peCode
Copy link
Author

Still no updates I still can not use threads in my electron app :(

@infFenrir
Copy link

I tested it on - Electron 4, 5, 6 and even 7 ....
The result is the same, app crash :(

@i7soft
Copy link

i7soft commented Jul 27, 2019

i have this problem too

@d0peCode
Copy link
Author

d0peCode commented Aug 3, 2019

Is there any chance of some statement from electron team? It's been months now (half year #17030)

@SaltFish001
Copy link

04(35B$JGE3MHE)_(S5M$G9
Add NODE_OPTIONS(--experimental-worker) in your system environment variable.Restart terminal.

@walmat
Copy link

walmat commented Aug 11, 2019

@SaltFish001 And what happens when you try to spawn a worker?

@SaltFish001
Copy link

@SaltFish001 And what happens when you try to spawn a worker?

It's work.

@SaltFish001
Copy link

SaltFish001 commented Aug 11, 2019

@SaltFish001 And what happens when you try to spawn a worker?

sorry, it's seems no working.The process ended and no error was reported

@d0peCode
Copy link
Author

I've tried it before even posting this issue. It doesn't change anything. Experimental flag for worker module was only required in node v11 if I remember correctly.

@coltorchen
Copy link

coltorchen commented Aug 21, 2019

just require('worker_threads') in render.js
const { isMainThread, parentPort, workerData, threadId, MessageChannel, MessagePort, Worker } = require('worker_threads');
image
I have the same problem about this.

this error seems throwed by node_work.cc, when you try to new Worker object in javascript.
it try to check env->isolate_data()->platform(),just like this.
image

@codebytere
Copy link
Member

codebytere commented Oct 3, 2019

Working on this now - just got #20416 up as a partial starter fix :)

@durasj

This comment has been minimized.

@codebytere
Copy link
Member

codebytere commented Oct 3, 2019

@durasj oh derp yes thanks, updated!

@codebytere codebytere self-assigned this Oct 3, 2019
@codebytere
Copy link
Member

codebytere commented Oct 12, 2019

Small update - still working on this!

it's related-but-different-than the ELECTRON_RUN_AS_NODE issue i put up a PR for earlier. This is related to the way we initialize Node.js in renderer processes and will require some changes in our renderer client code which will take a bit longer but for which i'll hopefully have more updates soon.

Thanks for your patience, all 😁

@linonetwo
Copy link

linonetwo commented Jul 19, 2020

I manage to make it work during dev and local production mode.

I can use ./dist/mac/TiddlyGit.app without error.

But app in ./dist/TiddlyGit-0.0.9.dmg resulting in the following error:

{"code":"MODULE_NOT_FOUND","message":"Cannot find module 'tiddlywiki'","requireStack":["/Applications/TiddlyGit.app/Contents/wiki-worker.js"],"level":"error"}

Very strange, seems compressing things into dmg breaks something.

I have tried electron 8 and electron 9.1.0

Seems in "real" production mode, script in worker_thread can't use electron's require hook, so it can't find dependences in asar.

@dsteinman
Copy link

Seems in "real" production mode, script in worker_thread can't use electron's require hook, so it can't find dependences in asar.

Without looking very closely, this sounds very similar to a problem I encountered last week here

In an electron packaged app, the app.asar file gets unpacked to a directory name app.asar-unpacked. So in this case, in production mode, the path of a file being loaded needed to have a search & replace performed to use the unpacked directory:

This might be what your tiddlywiki module needs. If it is loading something from a local path, you might be able to do something like this to get it to work:

const pathToFile = '.../myfile';
const unpackedPath = pathToFile.replace("app.asar", "app.asar.unpacked");
const module = require(pathToFile);

@linonetwo
Copy link

linonetwo commented Jul 19, 2020

I tried asar: false, and it just works. Seems the problem is that script in worker_threads can't require modules in asar.

@dsteinman tiddlywiki is a package in the node_modules, it is a self-contained 3rd party module, can only be required by name.


My Solution:

I'm using tiddlywiki, lodash and chokidar in my scripts, so:

electron-builder

    asarUnpack: [
      '**/node_modules/tiddlywiki/**/*',
      '**/node_modules/lodash/**/*',
      '**/node_modules/chokidar/**/*',
      // dep of git.js
      '**/node_modules/fs-extra/**/*',
      '**/node_modules/dugite/**/*',
      // dep of fs-extra
      '**/node_modules/jsonfile/**/*',
      '**/node_modules/universalify/**/*',
      '**/node_modules/graceful-fs/**/*',
      '**/node_modules/at-least-node/**/*',
      // dep of dugite
      '**/node_modules/rimraf/**/*',
      '**/node_modules/progress/**/*',
      '**/node_modules/mkdirp/**/*',
      '**/node_modules/minimist/**/*',
      '**/node_modules/glob/**/*',
      '**/node_modules/checksum/**/*',
      '**/node_modules/got/**/*',
      '**/node_modules/tar/**/*',
      '**/node_modules/fs.realpath/**/*',
      '**/node_modules/inflight/**/*',
      '**/node_modules/path-is-absolute/**/*',
      '**/node_modules/optimist/**/*',
      '**/node_modules/minimatch/**/*',
      '**/node_modules/inherits/**/*',
      '**/node_modules/@sindresorhus/is/**/*',
      '**/node_modules/@szmarczak/http-timer/**/*',
      '**/node_modules/decompress-response@/**/*',
      '**/node_modules/duplexer3/**/*',
      '**/node_modules/cacheable-request/**/*',
      '**/node_modules/get-stream/**/*',
      '**/node_modules/lowercase-keys/**/*',
      '**/node_modules/mimic-response/**/*',
      '**/node_modules/p-cancelable/**/*',
      '**/node_modules/to-readable-stream/**/*',
      '**/node_modules/url-parse-lax/**/*',
      '**/node_modules/fs-minipass/**/*',
      '**/node_modules/chownr/**/*',
      '**/node_modules/safe-buffer/**/*',
      '**/node_modules/once/**/*',
      '**/node_modules/minizlib/**/*',
      '**/node_modules/wrappy/**/*',
      '**/node_modules/wordwrap/**/*',
      '**/node_modules/defer-to-connect/**/*',
      '**/node_modules/minipass/**/*',
      '**/node_modules/clone-response/**/*',
      '**/node_modules/get-stream/**/*',
      '**/node_modules/http-cache-semantics/**/*',
      '**/node_modules/keyv/**/*',
      '**/node_modules/lowercase-keys/**/*',
      '**/node_modules/brace-expansion/**/*',
      '**/node_modules/responselike/**/*',
      '**/node_modules/pump/**/*',
      '**/node_modules/normalize-url/**/*',
      '**/node_modules/yallist/**/*',
      '**/node_modules/json-buffer/**/*',
      '**/node_modules/balanced-match/**/*',
      '**/node_modules/concat-map/**/*',
      '**/node_modules/prepend-http/**/*',
      '**/node_modules/end-of-stream/**/*',
      // dep of chokidar
      '**/node_modules/is-binary-path/**/*',
      '**/node_modules/normalize-path/**/*',
      '**/node_modules/glob-parent/**/*',
      '**/node_modules/braces/**/*',
      '**/node_modules/anymatch/**/*',
      '**/node_modules/is-glob/**/*',
      '**/node_modules/readdirp/**/*',
      '**/node_modules/fsevents/**/*',
      '**/node_modules/binary-extensions/**/*',
      '**/node_modules/picomatch/**/*',
      '**/node_modules/is-extglob/**/*',
      '**/node_modules/fill-range/**/*',
      '**/node_modules/to-regex-range/**/*',
      '**/node_modules/is-number/**/*',
    ],
    extraResources: [
      {
        from: 'public/libs/wiki/wiki-worker.js',
        to: 'app.asar.unpacked/wiki-worker.js',
      },
      {
        from: 'public/libs/wiki/watch-wiki-worker.js',
        to: 'app.asar.unpacked/watch-wiki-worker.js',
      },
      {
        from: 'public/libs/git.js',
        to: 'app.asar.unpacked/git.js',
      },
    ],

wow, our watch-wiki-worker.js dependes on my local git.js and git.js depends on chokidar and dugite.

seems chokidar and dugite depends on tons of things, and things depend on things... So I have to use npx npm-remote-ls --flatten chokidar -d false -o false and so on to get the flattened dependency tree...

If you miss one of the dep of dep, you will get MODULE_NOT_FOUND error too.

worker manager

const WIKI_WORKER_PATH = isDev
  ? path.resolve(__dirname, './wiki-worker.js')
  : path.resolve(process.resourcesPath, 'app.asar.unpacked', 'wiki-worker.js');
const WIKI_WATCHER_WORKER_PATH = isDev
  ? path.resolve(__dirname, './watch-wiki-worker.js')
  : path.resolve(process.resourcesPath, 'app.asar.unpacked', 'watch-wiki-worker.js');
  1. list all 3rd party package that is required in worker_threads script, in the asarUnpack
  2. use extraResources to copy worker_threads scripts and local files that are required in worker_threads script, to app.asar.unpacked

linonetwo added a commit to tiddly-gittly/TidGi-Desktop that referenced this issue Jul 19, 2020
@LBRGeorge
Copy link

I've made it works with Electron 9.x.x, however something weird is happening. The thread is making some sqlite queries and some intensive CPU tasks, until here is fine, however, when all the tasks finishes and the thread terminates (which should happen), the whole electron app simply crash.

@linonetwo
Copy link

@LBRGeorge I run into this once, where I set

worker.on('exit', code => {
    if (code !== 0)
      logger.warn(
        `WikiWatcher ${wikiRepoPath} Worker stopped with exit code ${code}, this also happen normally when you delete a workspace.`,
        loggerMeta,
      );

Electron just crush and I see the log get uncaughtException: logger.warn is not a function, actually I should use logger.warning, and this crush the electron.

You can try winston's exceptionHandlers, it can help you find the bug.

@codebytere
Copy link
Member

Hey folks - i understand that there's still active discussion in this thread, and insofar as it helps folks i encourage it to continue. However, this issue was opened as a reflection of the fact that worker_threads don't function in the renderer process, and a length investigation results in that being a problem whose solution is fundamentally incompatible with the architecture of the renderer process. I'm happy to provide more details if that would help contextualize this decision, but for now folks should expect moving forward that we will support Web Workers in the renderer to accomplish similar goals, and to use and open issues related to those instead of the worker_threads module provided by Node.js. If issues arise with regard to worker_threads in the main process we are also happy to investigate those as future individual issues.

@User0111
Copy link

I tried asar: false, and it just works. Seems the problem is that script in worker_threads can't require modules in asar.

@dsteinman tiddlywiki is a package in the node_modules, it is a self-contained 3rd party module, can only be required by name.

My Solution:

I'm using tiddlywiki, lodash and chokidar in my scripts, so:

electron-builder

    asarUnpack: [
      '**/node_modules/tiddlywiki/**/*',
      '**/node_modules/lodash/**/*',
      '**/node_modules/chokidar/**/*',
      // dep of git.js
      '**/node_modules/fs-extra/**/*',
      '**/node_modules/dugite/**/*',
      // dep of fs-extra
      '**/node_modules/jsonfile/**/*',
      '**/node_modules/universalify/**/*',
      '**/node_modules/graceful-fs/**/*',
      '**/node_modules/at-least-node/**/*',
      // dep of dugite
      '**/node_modules/rimraf/**/*',
      '**/node_modules/progress/**/*',
      '**/node_modules/mkdirp/**/*',
      '**/node_modules/minimist/**/*',
      '**/node_modules/glob/**/*',
      '**/node_modules/checksum/**/*',
      '**/node_modules/got/**/*',
      '**/node_modules/tar/**/*',
      '**/node_modules/fs.realpath/**/*',
      '**/node_modules/inflight/**/*',
      '**/node_modules/path-is-absolute/**/*',
      '**/node_modules/optimist/**/*',
      '**/node_modules/minimatch/**/*',
      '**/node_modules/inherits/**/*',
      '**/node_modules/@sindresorhus/is/**/*',
      '**/node_modules/@szmarczak/http-timer/**/*',
      '**/node_modules/decompress-response@/**/*',
      '**/node_modules/duplexer3/**/*',
      '**/node_modules/cacheable-request/**/*',
      '**/node_modules/get-stream/**/*',
      '**/node_modules/lowercase-keys/**/*',
      '**/node_modules/mimic-response/**/*',
      '**/node_modules/p-cancelable/**/*',
      '**/node_modules/to-readable-stream/**/*',
      '**/node_modules/url-parse-lax/**/*',
      '**/node_modules/fs-minipass/**/*',
      '**/node_modules/chownr/**/*',
      '**/node_modules/safe-buffer/**/*',
      '**/node_modules/once/**/*',
      '**/node_modules/minizlib/**/*',
      '**/node_modules/wrappy/**/*',
      '**/node_modules/wordwrap/**/*',
      '**/node_modules/defer-to-connect/**/*',
      '**/node_modules/minipass/**/*',
      '**/node_modules/clone-response/**/*',
      '**/node_modules/get-stream/**/*',
      '**/node_modules/http-cache-semantics/**/*',
      '**/node_modules/keyv/**/*',
      '**/node_modules/lowercase-keys/**/*',
      '**/node_modules/brace-expansion/**/*',
      '**/node_modules/responselike/**/*',
      '**/node_modules/pump/**/*',
      '**/node_modules/normalize-url/**/*',
      '**/node_modules/yallist/**/*',
      '**/node_modules/json-buffer/**/*',
      '**/node_modules/balanced-match/**/*',
      '**/node_modules/concat-map/**/*',
      '**/node_modules/prepend-http/**/*',
      '**/node_modules/end-of-stream/**/*',
      // dep of chokidar
      '**/node_modules/is-binary-path/**/*',
      '**/node_modules/normalize-path/**/*',
      '**/node_modules/glob-parent/**/*',
      '**/node_modules/braces/**/*',
      '**/node_modules/anymatch/**/*',
      '**/node_modules/is-glob/**/*',
      '**/node_modules/readdirp/**/*',
      '**/node_modules/fsevents/**/*',
      '**/node_modules/binary-extensions/**/*',
      '**/node_modules/picomatch/**/*',
      '**/node_modules/is-extglob/**/*',
      '**/node_modules/fill-range/**/*',
      '**/node_modules/to-regex-range/**/*',
      '**/node_modules/is-number/**/*',
    ],
    extraResources: [
      {
        from: 'public/libs/wiki/wiki-worker.js',
        to: 'app.asar.unpacked/wiki-worker.js',
      },
      {
        from: 'public/libs/wiki/watch-wiki-worker.js',
        to: 'app.asar.unpacked/watch-wiki-worker.js',
      },
      {
        from: 'public/libs/git.js',
        to: 'app.asar.unpacked/git.js',
      },
    ],

wow, our watch-wiki-worker.js dependes on my local git.js and git.js depends on chokidar and dugite.

seems chokidar and dugite depends on tons of things, and things depend on things... So I have to use npx npm-remote-ls --flatten chokidar -d false -o false and so on to get the flattened dependency tree...

If you miss one of the dep of dep, you will get MODULE_NOT_FOUND error too.

worker manager

const WIKI_WORKER_PATH = isDev
  ? path.resolve(__dirname, './wiki-worker.js')
  : path.resolve(process.resourcesPath, 'app.asar.unpacked', 'wiki-worker.js');
const WIKI_WATCHER_WORKER_PATH = isDev
  ? path.resolve(__dirname, './watch-wiki-worker.js')
  : path.resolve(process.resourcesPath, 'app.asar.unpacked', 'watch-wiki-worker.js');
  1. list all 3rd party package that is required in worker_threads script, in the asarUnpack
  2. use extraResources to copy worker_threads scripts and local files that are required in worker_threads script, to app.asar.unpacked

This solution works for me as well

I was not able to use worker_threads with the following configuration:
image

I've used Electron Workers described here. Tasks processing is x2 faster now. I had a lot of sex love with Electron before I made it works but it is doable. it was really hard to get the list of all the dependencies(50) I have to add to asarUnpack. It's weird that so powerfull feature not documented well and needs so a lot of tricks to do. I would be happy to get know that I can do something simpler to make it works in another way even after all this hell.

@linonetwo
Copy link

linonetwo commented May 23, 2021

@User0111 Actually I move to electron-forge, and use threads-plugin, so webpack pack standalone js file for worker to run, and it works fine and no asar-unpack now.

https://github.com/tiddly-gittly/TiddlyGit-Desktop/blob/afacc142f65b2afd951edd25ee4b731b6da3ad3f/webpack.plugins.js#L9-L41

https://github.com/tiddly-gittly/TiddlyGit-Desktop/blob/afacc142f65b2afd951edd25ee4b731b6da3ad3f/src/services/wiki/index.ts#L86

But it takes me a long time refactor my code to electron-forge + typescript.


And maybe it is because I upgraded to electron@12.0.6 ?

@m4heshd
Copy link

m4heshd commented Jun 24, 2021

Electron really need to add more support for worker_threads especially since it's a god given for GUI development. I still find it very weird why the worker is unable to use Electron's patched fs. The most of my time I've spent debugging my projects, is related to Electron + asar + worker threads. There's always a "gotcha" waiting around the corner when going into production. @linonetwo I see that you're going through the same exact hell as I am since I see you on every worker_threads related issue that I'm also having.

@linonetwo
Copy link

linonetwo commented Jun 24, 2021

@m4heshd Yes, but I finally make it worked... My latest solution includes:

  1. Use electron forge (webpack typescript) https://www.electronforge.io/templates/typescript-+-webpack-template
  2. copy thread-used-npm-packages to dist folder https://github.com/tiddly-gittly/TiddlyGit-Desktop/blob/283a3263792d33a200500d1d73b1cf821d41b6f0/webpack.plugins.js#L23-L29
  3. use threads plugin Webpack5 andywer/threads-plugin#43 (comment)

@m4heshd
Copy link

m4heshd commented Jun 24, 2021

I finally make it worked...

That's great. Unfortunately I cannot afford the time for any major refactoring right now. I'm just using my workers and dependencies unpacked for the moment. But love to move to a solid solution in the near future.

@Coukaratcha
Copy link

@linonetwo Thank you for your help. I have used the asar.unpacked trick and it worked like a charm. However, I am very disappointed that worker_threads are not working properly out of the box when the app is packaged.
I see that you found a better solution with electron-forge but still I think it should be great that electron improves worker_threads integration. I am not satisfied with @codebytere's reply as I think that the issue was misunderstood: the issue is not that worker_threads are not working in renderer process, but that it cannot be possible to load modules from worker when using worker_threads from the main process. I think unpacking modules, even if it is a clever idea and a good temporary workaround, is not a good solution as we lost the perks of packaging.
Is there any better solution if we want to keep nsis build target?

@JaspreetChhabra
Copy link

JaspreetChhabra commented May 21, 2022

@linonetwo Thanks for the help. The mentioned solution did work but my worker is also using 'electron-store' which internally uses 'electron' Module. 'electron' needs to be mentioned in the dev-dependecies and I am not able to unpack the modules mentioned in the dev-dependencies using "asarUnpack" in package.json. Any workaround for this ?
I am getting below error

`error Error: Cannot find module 'electron'
Require stack:

  • /Volumes/app 1.0.4/app.app/Contents/Resources/app.asar.unpacked/node_modules/electron-store/index.js`

@linonetwo
Copy link

@JaspreetChhabra  I think code in worker don't have access to electron modules, because electron module only accessible from electron main thread (not any worker thread).

A workaround may be passing signal to trigger things in main thread and return data back to the worker thread. You may use threads plugin to simplify this, it supports rxjs, so passing things here and there is not that terrible.

I think unpacking modules, even if it is a clever idea and a good temporary workaround, is not a good solution as we lost the perks of packaging.

@Coukaratcha I'm currently not unpacking everything, I only unpack as need. https://github.com/tiddly-gittly/TidGi-Desktop/blob/master/docs/Development.md#how-to-add-dependency-that-used-in-a-worker_thread

https://github.com/tiddly-gittly/TidGi-Desktop/blob/7831fa44bd27c18e1c25e62d78383d88c2930528/scripts/afterPack.js#L35-L50

It makes building complex a bit, but it works and not increasing too much bundle size.

well... 74M after install, but when zipped, no much , I think... See https://github.com/tiddly-gittly/TidGi-Desktop/releases TidGi-darwin-x64-0.7.7.zip 119 MB

@Cmaxd
Copy link

Cmaxd commented Apr 17, 2024

@linonetwo thanks for your solution. do you know if electron has made any progress on this matter? It has been two years

@linonetwo
Copy link

I don't think electron will do anything new on this, because worker_thread is working with simple npm dependency.

It depends on the npm package, if it use fs and path to locate other file, you have to move them to resource/ folder in afterPack script. I'm always doing this.

But I'm not sure now, because @giladgd say it is possible to locate .node dependency without manually move them? withcatai/node-llama-cpp#199 (comment)
I'm currently only move .node files to resource/ folder in afterBuild.js, and other JS dependency are bundled by electron-forge and works correctly in worker_thread.

@Cmaxd
Copy link

Cmaxd commented Apr 17, 2024

@linonetwo Thanks for reply. In my case, I have an SDK, the SDK calls a .dll based on ffi-napi, I configured SDK externals in webpack, and move the SDK to asar.unpacked folder, which runs well in main thread;
But now I want to call this SDK in worker_threads, it report an error MODULE_NOT_FOUND, to solve this problem, I need to bundle the SDK and worker's JS code together? and move the .node file in SDK's dependencies(ffi-napi ... ) to asar.unpacked folder ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
5.0.x
Unsorted Issues
Development

No branches or pull requests