FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

[stu31]

Hello,

We're trying to migrate to Serverless. Unfortunately, it looks like sls package -s dev just dies. We have a large project consisting of over 50 microservices. It seems that webpack is the issue when Googling, but no one really knows, nor does anyone have any good documentation on how to resolve this issue. If this has already been solved, please add a comment in the readme, so others can use this exciting tool.

... Serverless: Including resource definition: cloudformation/health_check_failed.js Serverless: WARNING: Resource Mappings has multiple definitions. They will be merged.) Serverless: Including resource definition: cloudformation/params.js Serverless: Bundling with Webpack...

<--- Last few GCs --->

[22459:0x102654000] 70144 ms: Mark-sweep 1124.7 (1448.7) -> 1124.7 (1443.2) MB, 418.2 / 0.0 ms (average mu = 0.030, current mu = 0.000) allocation failure GC in old space requested
[22459:0x102654000] 70664 ms: Mark-sweep 1132.0 (1449.3) -> 1131.0 (1442.8) MB, 493.3 / 0.0 ms (average mu = 0.040, current mu = 0.051) allocation failure GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

0: ExitFrame [pc: 0x187ce5a5be3d]

Security context: 0x0049a491e6e9
1: byteLength(aka byteLength) [0x495a312c39] [buffer.js:531] [bytecode=0x49a3359161 offset=204](this=0x0049108826f1 ,string=0x0049f4851bc9 <Very long string[3131506]>,encoding=0x0049a493e991 <String[4]: utf8>)
2: arguments adaptor frame: 3->2
3: fromString(aka fromString) [0x495a320709] [buffer.js:342] [bytecode=0x49a3353141 offset=74]...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
1: 0x10003c597 node::Abort() [/usr/local/bin/node]
2: 0x10003c7a1 node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
3: 0x1001ad575 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
4: 0x100579242 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
5: 0x10057bd15 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/local/bin/node]
6: 0x100577bbf v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
7: 0x100575d94 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
8: 0x10058262c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
9: 0x1005826af v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/local/bin/node]
10: 0x100554534 v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/local/bin/node]
11: 0x100686109 v8::internal::String::SlowFlatten(v8::internal::Handlev8::internal::ConsString, v8::internal::PretenureFlag) [/usr/local/bin/node]
12: 0x1001cb30d v8::String::Utf8Length() const [/usr/local/bin/node]
13: 0x100050303 node::Buffer::(anonymous namespace)::ByteLengthUtf8(v8::FunctionCallbackInfov8::Value const&) [/usr/local/bin/node]
14: 0x10023663f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
15: 0x100235b81 v8::internal::MaybeHandlev8::internal::Object v8::internal::(anonymous namespace)::HandleApiCallHelper(v8::internal::Isolate*, v8::internal::Handlev8::internal::HeapObject, v8::internal::Handlev8::internal::HeapObject, v8::internal::Handlev8::internal::FunctionTemplateInfo, v8::internal::Handlev8::internal::Object, v8::internal::BuiltinArguments) [/usr/local/bin/node]
16: 0x100235220 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
17: 0x187ce5a5be3d
Abort trap: 6
serverless@1.49.0
│ ├─┬ @serverless/cli@1.0.1
│ │ ├─┬ @serverless/core@1.0.0

[medikoo]

It's OOM error, which technically means that something tries to process too much within process memory.

Do you have an idea what characteristics of your app may cause it (?) Have you tried to deploy the service without serverless-webpack plugin?

[pmuens]

You might also want to look into https://github.com/FormidableLabs/serverless-jetpack

[herebebogans]

Had similar issues with largish typescript projects and serverless-webpack-plugin.

Try

export NODE_OPTIONS=--max_old_space_size=8192

[stu31]

@herebebogans That did the trick!

[jimcamut]

@herebebogans I'm having a similar issue, which might be due to serverless-webpack-plugin. Where do you add export NODE_OPTIONS=--max_old_space_size=8192 to?

[dundas]

@herebebogans I'm having a similar issue, which might be due to serverless-webpack-plugin. Where do you add export NODE_OPTIONS=--max_old_space_size=8192 to?

In the command line in root of your project.

[stu31]

Dundas, You would probably want to add that to a shell script, if I’m understanding your question correctly. #deploy.sh export NODE_OPTIONS=--max_old_space_size=8192 sls deploy Whatever shell process you are running in, setting these values before running serverless deploy; your node environment should pick that up. In our case, it seems webpack was causing this error, so I believe this should prevent this error. You could always play around with that value and see what works for you. We have quite a few microservices that get deployed in one template. If you’re using containers, then you would probably need to set your dockerfiles environment variables and rebuild, then rerun your deploy script, since node will be running in a separate process. From: dundas <notifications@github.com> Reply-To: serverless/serverless <reply@reply.github.com> Date: Tuesday, September 3, 2019 at 5:09 PM To: serverless/serverless <serverless@noreply.github.com> Cc: Stu Murry <stu@stumurry.com>, Author <author@noreply.github.com> Subject: Re: [serverless/serverless] FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory (#6503) @herebebogans<https://github.com/herebebogans> I'm having a similar issue, which might be due to serverless-webpack-plugin. Where do you add export NODE_OPTIONS=--max_old_space_size=8192 to? In the command line in root of your project. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#6503?email_source=notifications&email_token=ALF3MHWNNJWJYJ5WM3IPWUDQH3VCRA5CNFSM4IJMIHOKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5Z2ZBA#issuecomment-527674500>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ALF3MHXVYFJOOVJL6VJOMZ3QH3VCRANCNFSM4IJMIHOA>.

[stefanKuijers]

@revgum : You referenced a comment as a solution to your issue but @rafa-as stated that this does not work if I understand well. I think it would be helpfull to the community if you mention this on that issue as well ;). Maybe the two of you could figure out what the differences are between your approaches

[revgum]

@revgum : You referenced a comment as a solution to your issue but @rafa-as stated that this does not work if I understand well. I think it would be helpfull to the community if you mention this on that issue as well ;). Maybe the two of you could figure out what the differences are between your approaches

You're right, it turns out I was mistakenly thinking it worked.. but testing it again proved otherwise. 🤔

[stefanKuijers]

In that case we can come to the same conclusion:

however promising the npmrc solution sounds, like mentioned in nodejs/help#1783 (comment), It does not work yet.

I tried some tests this morning and even though there is enough free memory, the process fails far before the memory usage reaches the 4gb I set it to in npmrc.

[stefanKuijers]

Forgot to mention, for the sake of clarity, that running the script with those node options does work out well.

node --max-old-space-size=4076 script.js

But as others mentioned already: that's not a pretty solution. Adding it in .npmrc would be prefereed.

[bcraft]

I have been working on a serverless project for a while, approximately 50 functions in typescript and was always building on a server. I have now moved to a machine with 16gb ram and running serverless webpack uses the entire system memory leading to a crash. I am not packing any external modules, just function code.

It does seem there is a memory leak, the error appears to be coming from a service worker in the spawn process. Would a possible solution be to limit the number of concurrent subprocesses launched at once?

[Nightbr]

Hey, still having this issue:

<--- Last few GCs --->

[89788:0x2e78f70]    25018 ms: Mark-sweep 2036.4 (2049.2) -> 2035.7 (2049.2) MB, 239.8 / 0.1 ms  (average mu = 0.145, current mu = 0.000) last resort GC in old space requested
[89788:0x2e78f70]    25352 ms: Mark-sweep 2039.1 (2051.1) -> 2039.2 (2052.1) MB, 263.1 / 0.1 ms  (average mu = 0.178, current mu = 0.211) allocation failure GC in old space requested


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x0680ba6408d1 <JSObject>
    0: builtin exit frame: compileFunction(this=0x2ef8acc43161 <JSGlobal Object>,0x3a8162e1a301 <JSArray[5]>,0x3a8162e1a2e1 <JSArray[0]>,0x3321332c04b1 <undefined>,0x3321332c06e9 <false>,0x3321332c04b1 <undefined>,0,0,0x1fe1d6d66ce1 <String[#98]: /home/nightbringer/Dev/efounders/filex/apps/core-engine/index-file/.webpack/service/src/handler.js>,0x06f3f32c0119 <Very long string[31266088]>,0x2ef8a...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0xa18150 node::Abort() [node]
 2: 0xa1855c node::OnFatalError(char const*, char const*) [node]
 3: 0xb9705e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xb973d9 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xd54655  [node]
 6: 0xd54ce6 v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [node]
 7: 0xd615a5 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xd62455 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xd64f0c v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [node]
10: 0xd344ae v8::internal::Factory::NewScopeInfo(int, v8::internal::AllocationType) [node]
11: 0xf63091 v8::internal::ScopeInfo::Create(v8::internal::Isolate*, v8::internal::Zone*, v8::internal::Scope*, v8::internal::MaybeHandle<v8::internal::ScopeInfo>) [node]
12: 0xbf4338 v8::internal::Scope::AllocateScopeInfosRecursively(v8::internal::Isolate*, v8::internal::MaybeHandle<v8::internal::ScopeInfo>) [node]
13: 0xbf4441 v8::internal::DeclarationScope::AllocateScopeInfos(v8::internal::ParseInfo*, v8::internal::Isolate*) [node]
14: 0xc73b6b  [node]
15: 0xc7603c v8::internal::Compiler::GetWrappedFunction(v8::internal::Handle<v8::internal::String>, v8::internal::Handle<v8::internal::FixedArray>, v8::internal::Handle<v8::internal::Context>, v8::internal::Compiler::ScriptDetails const&, v8::ScriptOriginOptions, v8::internal::ScriptData*, v8::ScriptCompiler::CompileOptions, v8::ScriptCompiler::NoCacheReason) [node]
16: 0xbabd36 v8::ScriptCompiler::CompileFunctionInContext(v8::Local<v8::Context>, v8::ScriptCompiler::Source*, unsigned long, v8::Local<v8::String>*, unsigned long, v8::Local<v8::Object>*, v8::ScriptCompiler::CompileOptions, v8::ScriptCompiler::NoCacheReason, v8::Local<v8::ScriptOrModule>*) [node]
17: 0xa0badd node::contextify::ContextifyContext::CompileFunction(v8::FunctionCallbackInfo<v8::Value> const&) [node]
18: 0xc02fc9  [node]
19: 0xc04db7 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [node]
20: 0x140a8f9  [node]
Aborted (core dumped)
ERROR: Something went wrong in @nrwl/run-commands - Command failed: sls offline

With this plugins & version:

frameworkVersion: ^2.28.3
plugins:
  - serverless-bundle
  - serverless-offline-sqs
  - serverless-offline
useDotenv: true 
# serverless optimization
package:
  individually: true

custom:
  bundle:
    sourcemaps: true
    tsConfig: "tsconfig.app.json"
    linting: true
    packager: yarn

Try with NODE_OPTIONS="--max_old_space_size=4096" but it crashs my computer (32Go RAM) so there is a memory leak somewhere.

It keeps rise at each lambda execution & will crash after reaching the max_old_space_size.

EDIT: seems it come from serverless-offline -> dherault/serverless-offline#1119 (comment)

[Southclaws]

Also running into this while packaging up a fairly small one-function project. No plugins and no webpack, just some javascript code. Unsure how to investigate or resolve tbh.

[medikoo]

Moving it into discussions, as it's related to specific plugins usage, and it's not an issue in Framework core