V8 provides [v8::Isolate::AddNearHeapLimitCallback()](https://source.chromium.org/chromium/chromium/src/+/master:v8/include/v8.h;bpv=1;bpt=1;l=9343?originalUrl=https:%2F%2Fcs.chromium.org%2F&gsn=AddNearHeapLimitCallback&gs=kythe%3A%2F%2Fchromium.googlesource.com%2Fchromium%2Fsrc%3Flang%3Dc%252B%252B%3Fpath%3Dsrc%2Fv8%2Finclude%2Fv8.h%23RJGJDE44AxlIHgLihqvHnOXXbWkTVPQNjeG939U-FYQ) for adjusting the heap limit when V8 is approaching it. The debugger implementation in V8 uses this to break at the point where the heap size limit is near in the Chrome DevTools. I did a proof of concept --heapsnapshot-near-heap-limit implementation (https://github.com/nodejs/node/pull/33010), it does work if I just write a snapshot to disk in that callback. I temporarily raise the heap size limit to a value slightly bigger than the original limit until the snapshot is done, and tells V8 to restore to the initial limit later. There are some observations with this approach:

[Leaking]

V8 provides v8::Isolate::AddNearHeapLimitCallback() for adjusting the heap limit when V8 is approaching it. The debugger implementation in V8 uses this to break at the point where the heap size limit is near in the Chrome DevTools. I did a proof of concept --heapsnapshot-near-heap-limit implementation (nodejs/node#33010), it does work if I just write a snapshot to disk in that callback. I temporarily raise the heap size limit to a value slightly bigger than the original limit until the snapshot is done, and tells V8 to restore to the initial limit later. There are some observations with this approach:

• Once the heap limit is raised, V8 can't restore the heap limit back to the initial one until the heap usage fall back under the initial limit. So for a heap with unbounded growth, if we set the new limit to 2x, this means the process will eventually crash with 2x of the original heap limit, effectively raising the max heap size to 2x of the original setting. We could try some tricks to adjust this limit on-the-fly, I don't have any good ideas about this.
  • We could make that new limit configurable by user, of course.
  • Somehow just setting it to initial limit + 1 doesn't seem to lead to any issues, even for relatively big heaps. Maybe V8 doesn't really use the memory from the managed heap that much to do the snapshotting (I have not measured the RSS yet).
• When the process is approaching that limit, it could spend a lot of time writing snapshots - multiple ones, because the snapshot creation triggers GC eagerly, so the process will live a bit longer than usual, hitting the limit back and forth. Then compared to doing nothing at all, the process will stay irresponsive for a longer period of time before it crashes. For example, I tweaked the max limit to initial limit + 1, and with test/fixtures/workload/allocation.js and --max-old-space-size=100, without using --heapsnapshot-near-heap-limit, the process crashes in 20s after 73 GCs. With the option on it crashes in 121s after 130 GCs, leaving 12 snapshots of size 140-170MB on disks.
  • We could also limit the amount of snapshot we want to write with this option, to avoid keeping the process in limbo for longer than it should. This can also be configurable.
  • This actually should help with the issue raised in Create heapdump on out of memory nodejs/node#27552 (comment) - with multiple snapshots the users can find out what's growing inside the heap and with our existing snapshot naming machinery the files are aptly named in a way for you to locate them in order ( it goes like Heap.20200423.063523.40985.0.001.heapsnapshot, Heap.20200423.063524.40985.0.002.heapsnapshot..)

Any comments about these observations?

Originally posted by @joyeecheung in nodejs/node#27552 (comment)