worker: add ability to take heap snapshot from parent thread #31569

addaleax · 2020-01-29T16:46:35Z

@nodejs/workers

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

richardlau · 2020-01-29T17:17:05Z

Add a test for when the worker is no longer running?

addaleax · 2020-01-29T17:30:55Z

@richardlau Thanks, yes – I had missed that I hadn’t fully implemented that part yet 👍

miss-click sorry about that

doc/api/errors.md

doc/api/worker_threads.md

lib/internal/worker.js

lib/internal/heap_utils.js

lib/internal/errors.js

nodejs-github-bot · 2020-01-30T16:23:35Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28783/

nodejs-github-bot · 2020-01-31T21:15:58Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28821/

nodejs-github-bot · 2020-02-01T02:45:59Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28827/

Trott · 2020-02-01T03:11:20Z

Needs a rebase. Also is the failure of test.parallel/test-bootstrap-modules CI when run with tools/test.py --worker related to this change? I haven't seen it before and we got it three times in a row here.

richardlau · 2020-02-01T04:08:31Z

Needs a rebase. Also is the failure of test.parallel/test-bootstrap-modules CI when run with tools/test.py --worker related to this change? I haven't seen it before and we got it three times in a row here.

Yes, it is related:

21:56:18 not ok 112 parallel/test-bootstrap-modules
21:56:18   ---
21:56:18   duration_ms: 0.278
21:56:18   severity: fail
21:56:18   exitcode: 1
21:56:18   stack: |-
21:56:18     
21:56:18     events.js:298
21:56:18           throw er; // Unhandled 'error' event
21:56:18           ^
21:56:18     AssertionError [ERR_ASSERTION]: These modules were unexpectedly loaded:
21:56:18       Internal Binding stream_wrap,
21:56:18       Internal Binding uv,
21:56:18       NativeModule internal/heap_utils,
21:56:18       NativeModule internal/stream_base_commons
21:56:18     
21:56:18         at Object.<anonymous> (/home/iojs/build/workspace/node-test-commit-custom-suites-freestyle/test/parallel/test-bootstrap-modules.js:131:8)
21:56:18         at Module._compile (internal/modules/cjs/loader.js:1208:30)
21:56:18         at Object.Module._extensions..js (internal/modules/cjs/loader.js:1228:10)
...

test-bootstrap-modules is there to alert when extra modules are being loaded by the bootstrap process, which they are for workers in this PR, which has added internal/heap_utils and requires it from internal/worker. internal/heap_utils requires internal/stream_base_commons and that requires stream_wrap and uv (

node/lib/internal/stream_base_commons.js

Lines 10 to 18 in cb21011

    
           const { 
        
             WriteWrap, 
        
             kReadBytesOrError, 
        
             kArrayBufferOffset, 
        
             kBytesWritten, 
        
             kLastWriteWasAsync, 
        
             streamBaseState 
        
           } = internalBinding('stream_wrap'); 
        
           const { UV_EOF } = internalBinding('uv');

).

If we're okay adding these extra modules to the bootstrapping of worker threads (it's probably okay -- the whole point of the test is to make us make a considered decision) then they should be added to the list of modules expected to be loaded for workers in the test:

node/test/parallel/test-bootstrap-modules.js

Lines 86 to 107 in ab9e894

    
           if (!common.isMainThread) { 
        
             expectedModules.add('Internal Binding messaging'); 
        
             expectedModules.add('Internal Binding symbols'); 
        
             expectedModules.add('Internal Binding worker'); 
        
             expectedModules.add('NativeModule _stream_duplex'); 
        
             expectedModules.add('NativeModule _stream_passthrough'); 
        
             expectedModules.add('NativeModule _stream_readable'); 
        
             expectedModules.add('NativeModule _stream_transform'); 
        
             expectedModules.add('NativeModule _stream_writable'); 
        
             expectedModules.add('NativeModule internal/error-serdes'); 
        
             expectedModules.add('NativeModule internal/process/worker_thread_only'); 
        
             expectedModules.add('NativeModule internal/streams/buffer_list'); 
        
             expectedModules.add('NativeModule internal/streams/destroy'); 
        
             expectedModules.add('NativeModule internal/streams/end-of-stream'); 
        
             expectedModules.add('NativeModule internal/streams/legacy'); 
        
             expectedModules.add('NativeModule internal/streams/pipeline'); 
        
             expectedModules.add('NativeModule internal/streams/state'); 
        
             expectedModules.add('NativeModule internal/worker'); 
        
             expectedModules.add('NativeModule internal/worker/io'); 
        
             expectedModules.add('NativeModule stream'); 
        
             expectedModules.add('NativeModule worker_threads'); 
        
           }

otherwise perhaps internal/heap_utils should be lazily loaded.

codecov-io · 2020-02-01T04:19:31Z

Codecov Report

Merging #31569 into master will increase coverage by 1.13%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master   #31569      +/-   ##
==========================================
+ Coverage   96.16%    97.3%   +1.13%     
==========================================
  Files         193      193              
  Lines       64651    64653       +2     
==========================================
+ Hits        62172    62910     +738     
+ Misses       2479     1743     -736

Impacted Files	Coverage Δ
lib/_http_server.js	`98.16% <0%> (-0.12%)`	⬇️
lib/internal/process/per_thread.js	`98.62% <0%> (ø)`	⬆️
lib/internal/http2/core.js	`96.51% <0%> (+0.12%)`	⬆️
lib/internal/fs/utils.js	`97.2% <0%> (+0.29%)`	⬆️
lib/child_process.js	`100% <0%> (+0.57%)`	⬆️
lib/internal/child_process/serialization.js	`93.49% <0%> (+0.81%)`	⬆️
lib/internal/streams/async_iterator.js	`99.12% <0%> (+0.85%)`	⬆️
lib/internal/url.js	`98.68% <0%> (+1.17%)`	⬆️
lib/internal/child_process.js	`97.23% <0%> (+1.29%)`	⬆️
lib/net.js	`98.46% <0%> (+1.41%)`	⬆️
... and 26 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2272489...6bbff20. Read the comment docs.

addaleax · 2020-02-01T10:30:39Z

Needs a rebase.

Done! :)

otherwise perhaps internal/heap_utils should be lazily loaded.

Yeah, I think that makes sense here. Done! 👍

nodejs-github-bot · 2020-02-01T10:31:32Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28841/

nodejs-github-bot · 2020-02-01T22:51:24Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28866/

nodejs-github-bot · 2020-02-02T13:15:27Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28875/

nodejs-github-bot · 2020-02-02T23:37:42Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28881/

nodejs-github-bot · 2020-02-03T03:47:59Z

CI: https://ci.nodejs.org/job/node-test-pull-request/28887/

Trott · 2020-02-03T15:04:35Z

Landed in 875a4d1

PR-URL: #31569 Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

Notable changes: * async_hooks * add executionAsyncResource (Matteo Collina) #30959 * crypto * add crypto.diffieHellman (Tobias Nießen) #31178 * add DH support to generateKeyPair (Tobias Nießen) #31178 * simplify DH groups (Tobias Nießen) #31178 * add key type 'dh' (Tobias Nießen) #31178 * test * skip keygen tests on arm systems (Tobias Nießen) #31178 * perf_hooks * add property flags to GCPerformanceEntry (Kirill Fomichev) #29547 * process * report ArrayBuffer memory in `memoryUsage()` (Anna Henningsen) #31550 * readline * make tab size configurable (Ruben Bridgewater) #31318 * report * add support for Workers (Anna Henningsen) #31386 * worker * add ability to take heap snapshot from parent thread (Anna Henningsen) #31569 * added new collaborators * add ronag to collaborators (Robert Nagy) #31498 PR-URL: #31837

Adapt doc to match implementation which exports getHeapSnapshot(). PR-URL: #32061 Refs: #31569 Refs: https://github.com/nodejs/node/blob/987a67339518d0380177a2e589f2bbd274230d0e/lib/internal/worker.js#L323 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Shelley Vohr <codebytere@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

PR-URL: nodejs#31569 Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

Adapt doc to match implementation which exports getHeapSnapshot(). PR-URL: nodejs#32061 Refs: nodejs#31569 Refs: https://github.com/nodejs/node/blob/987a67339518d0380177a2e589f2bbd274230d0e/lib/internal/worker.js#L323 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Shelley Vohr <codebytere@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

PR-URL: #31569 Reviewed-By: Denys Otrishko <shishugi@gmail.com> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Gus Caplan <me@gus.host> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com> Reviewed-By: Rich Trott <rtrott@gmail.com>

Adapt doc to match implementation which exports getHeapSnapshot(). PR-URL: #32061 Refs: #31569 Refs: https://github.com/nodejs/node/blob/987a67339518d0380177a2e589f2bbd274230d0e/lib/internal/worker.js#L323 Reviewed-By: Anna Henningsen <anna@addaleax.net> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Shelley Vohr <codebytere@gmail.com> Reviewed-By: Matheus Marchini <mat@mmarchini.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Ruben Bridgewater <ruben@bridgewater.de>

addaleax added semver-minor PRs that contain new features and should be released in the next minor version. worker Issues and PRs related to Worker support. labels Jan 29, 2020

nodejs-github-bot added the lib / src Issues and PRs related to general changes in the lib or src directory. label Jan 29, 2020

addaleax removed the lib / src Issues and PRs related to general changes in the lib or src directory. label Jan 29, 2020

addaleax force-pushed the worker-heapdump branch from a1e2c47 to 60e6b4f Compare January 29, 2020 17:30

vdeturckheim previously approved these changes Jan 29, 2020

View reviewed changes

devsnek reviewed Jan 29, 2020

View reviewed changes

doc/api/errors.md Outdated Show resolved Hide resolved

doc/api/worker_threads.md Outdated Show resolved Hide resolved

lib/internal/worker.js Outdated Show resolved Hide resolved

jasnell approved these changes Jan 29, 2020

View reviewed changes

devsnek approved these changes Jan 29, 2020

View reviewed changes

cjihrig approved these changes Jan 30, 2020

View reviewed changes

lundibundi approved these changes Jan 30, 2020

View reviewed changes

lib/internal/heap_utils.js Show resolved Hide resolved

lib/internal/errors.js Show resolved Hide resolved

awibox approved these changes Jan 30, 2020

View reviewed changes

richardlau approved these changes Jan 31, 2020

View reviewed changes

Trott approved these changes Feb 1, 2020

View reviewed changes

addaleax added 6 commits February 1, 2020 11:15

worker: add ability to take heap snapshot from parent thread

406c024

fixup! worker: add ability to take heap snapshot from parent thread

7a7e76f

fixup! worker: add ability to take heap snapshot from parent thread

1500f9e

fixup! worker: add ability to take heap snapshot from parent thread

60bcee1

fixup! worker: add ability to take heap snapshot from parent thread

bc64b60

fixup! worker: add ability to take heap snapshot from parent thread

6bbff20

addaleax force-pushed the worker-heapdump branch from 9aae228 to 6bbff20 Compare February 1, 2020 10:30

addaleax added the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Feb 1, 2020

richardlau approved these changes Feb 1, 2020

View reviewed changes

Trott closed this Feb 3, 2020

addaleax deleted the worker-heapdump branch February 4, 2020 16:41

codebytere mentioned this pull request Feb 17, 2020

v13.9.0 proposal #31837

Merged

hyj1991 mentioned this pull request Feb 21, 2020

Worker threads hyj1991/v8-profiler-next#9

Closed

Flarna mentioned this pull request Mar 3, 2020

doc: change worker.takeHeapSnapshot to getHeapSnapshot #32061

Closed

3 tasks

targos mentioned this pull request Apr 20, 2020

src: revamp internal field count tracking #31960

Closed

2 tasks

targos removed the author ready PRs that have at least one approval, no pending requests for changes, and a CI started. label Apr 25, 2020

targos mentioned this pull request May 2, 2020

v12.17.0 release proposal #33197

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

worker: add ability to take heap snapshot from parent thread #31569

worker: add ability to take heap snapshot from parent thread #31569

addaleax commented Jan 29, 2020

richardlau commented Jan 29, 2020

addaleax commented Jan 29, 2020

nodejs-github-bot commented Jan 30, 2020

nodejs-github-bot commented Jan 31, 2020

nodejs-github-bot commented Feb 1, 2020

Trott commented Feb 1, 2020

richardlau commented Feb 1, 2020

codecov-io commented Feb 1, 2020 •

edited

addaleax commented Feb 1, 2020

nodejs-github-bot commented Feb 1, 2020

nodejs-github-bot commented Feb 1, 2020

nodejs-github-bot commented Feb 2, 2020

nodejs-github-bot commented Feb 2, 2020

nodejs-github-bot commented Feb 3, 2020

Trott commented Feb 3, 2020

worker: add ability to take heap snapshot from parent thread #31569

worker: add ability to take heap snapshot from parent thread #31569

Conversation

addaleax commented Jan 29, 2020

Checklist

richardlau commented Jan 29, 2020

addaleax commented Jan 29, 2020

nodejs-github-bot commented Jan 30, 2020

nodejs-github-bot commented Jan 31, 2020

nodejs-github-bot commented Feb 1, 2020

Trott commented Feb 1, 2020

richardlau commented Feb 1, 2020

codecov-io commented Feb 1, 2020 • edited

Codecov Report

addaleax commented Feb 1, 2020

nodejs-github-bot commented Feb 1, 2020

nodejs-github-bot commented Feb 1, 2020

nodejs-github-bot commented Feb 2, 2020

nodejs-github-bot commented Feb 2, 2020

nodejs-github-bot commented Feb 3, 2020

Trott commented Feb 3, 2020

codecov-io commented Feb 1, 2020 •

edited