citizen goes out-of-memory

[ericrichtert]

We run citizen on a EC2 instance, and when uploading a large module (a few bytes of terraform and 1 large jar file for a lambda), the citizen process will take a lot of CPU and memory.

Finally, we get this in the logs:

Jun 22 19:06:06 ip-10-110-185-69 citizen: <--- Last few GCs --->
Jun 22 19:06:06 localhost citizen: [1214:0x54136d0] 600792 ms: Mark-sweep 1863.5 (1866.9) -> 1862.6 (1866.9) MB, 2795.7 / 0.0 ms (average mu = 0.072, current mu = 0.014) allocation failure scavenge might not succeed
Jun 22 19:06:06 localhost citizen: [1214:0x54136d0] 602898 ms: Mark-sweep 1863.6 (1866.9) -> 1862.7 (1866.9) MB, 2080.7 / 0.0 ms (average mu = 0.045, current mu = 0.012) allocation failure scavenge might not succeed
Jun 22 19:06:06 localhost citizen: <--- JS stacktrace --->
Jun 22 19:06:06 localhost citizen: FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Jun 22 19:06:06 localhost citizen: 1: 0xa123d0 node::Abort() [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 2: 0x962467 node::FatalError(char const*, char const*) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 3: 0xb3878e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 4: 0xb38b07 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 5: 0xcd91e5 [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 6: 0xcd9b8b v8::internal::Heap::RecomputeLimits(v8::internal::GarbageCollector) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 7: 0xce7b42 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 8: 0xce8995 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 9: 0xceb35c v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 10: 0xcba83b v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 11: 0xfeb0ff v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/opt/citizen/citizen]
Jun 22 19:06:06 localhost citizen: 12: 0x13451f9 [/opt/citizen/citizen]
Jun 22 19:06:07 localhost systemd: citizen.service: main process exited, code=killed, status=6/ABRT
Jun 22 19:06:07 localhost systemd: Unit citizen.service entered failed state.
Jun 22 19:06:07 localhost systemd: citizen.service failed.

Is there a way to solve this?