Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Heap out of memory memoryLimit issue in ForkTsCheckerWebpackPlugin #271

Closed
marcusjwhelan opened this issue Apr 25, 2019 · 45 comments
Closed
Labels

Comments

@marcusjwhelan
Copy link

Current behavior

Upgraded to start using electron-webpack and upgraded all our packages to newer versions.

Expected behavior

Typescript and webpack should compile code

Steps to reproduce the issue

use electron webpack with typescript?

Issue reproduction repository

Environment

  • fork-ts-checker-webpack-plugin: [version from the package.json]
    in node modules under /electron-webpack-ts/package.json
    fork-ts-checker-webpack-plugin: 0.5.2
  • typescript: [version from the package.json]
    typescript: 3.4.5
  • tslint: [version from the package.json]
    "tslint": "^5.14.0",
  • webpack: [version from the package.json]
    "webpack": "4.28.4"
  • os: [e.g. Ubuntu 19.04]
    Ubuntu 18.04
<--- Last few GCs --->

[8366:0x3f04ad0]    23325 ms: Mark-sweep 2042.0 (2085.7) -> 2041.3 (2086.2) MB, 734.9 / 0.0 ms  (average mu = 0.102, current mu = 0.013) allocation failure scavenge might not succeed
[8366:0x3f04ad0]    24138 ms: Mark-sweep 2043.2 (2086.2) -> 2042.5 (2087.2) MB, 807.7 / 0.0 ms  (average mu = 0.055, current mu = 0.007) allocation failure scavenge might not succeed


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x309d6aa5be1d]
    1: StubFrame [pc: 0x309d6aa5d1df]
Security context: 0x124119b1e6e9 <JSObject>
    2: getIntersectionType(aka getIntersectionType) [0x1a0a8652b3a1] [/home/marcus/work/ClientElectron/node_modules/typescript/lib/typescript.js:~39985] [pc=0x309d6b65f1d2](this=0x0d5af00026f1 <undefined>,types=0x3352c04a57b9 <JSArray[23]>,aliasSymbol=0x0d5af00026f1 <undefined>,aliasTypeArguments=0x0d5af00026f1 <undefine...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc510 node::Abort() [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 2: 0x8dc55c  [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 5: 0xec7bf2  [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
10: 0xea07f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
11: 0x114018e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/marcus/.nvm/versions/node/v10.15.3/bin/node]
12: 0x309d6aa5be1d 

Error from running electron-webpack dev

┏ Renderer -------------------

  Type checking and linting aborted - probably out of memory. Check `memoryLimit` option in ForkTsCheckerWebpackPlugin configuration.

┗ ----------------------------
┏ Main -----------------------

  Type checking and linting aborted - probably out of memory. Check `memoryLimit` option in ForkTsCheckerWebpackPlugin configuration.

┗ ----------------------------

@johnnyreilly
Copy link
Member

Could be worth upping your memoryLimit

https://github.com/Realytics/fork-ts-checker-webpack-plugin/blob/master/README.md#options

@marcusjwhelan
Copy link
Author

@johnnyreilly I moved the memory to 8192 but still failed with the same error just takes a little longer. I believe after talking to someone from electron-webpack that my issue could be a memory leak having to do with Typescript itself.

@marcusjwhelan
Copy link
Author

@johnnyreilly I also made a simple repo with what I used. It works just fine. However as I add more reducers/Actions the application then crashes with the same error.

@phryneas
Copy link
Contributor

@marcusjwhelan could you share that repro so I could take a peek? (No guarantees though.) Sometimes there are just TS patterns that lead to a looot of memory consumption.

@marcusjwhelan
Copy link
Author

@phryneas electron-userland/electron-webpack#280
https://github.com/marcusjwhelan/testingproj

While using electron-webpack is how I get this issue. That link to that project, that project works just fine, however, as you add more redux/actions to the structure of the application it will memory leak out, No matter the memoryLimit set.

@phryneas
Copy link
Contributor

phryneas commented May 3, 2019

First thing I am noticing: you are including all .d.ts in node_modules which will include these in the typecheck - usually those should be automatically included as they are the default value of typeRoots. Does everything keep working when you remove them from include in the tsconfig.json? If not, try using compilerOptions.typeRoots instead of include for those.

PS: just invoking tsc --noEmit takes about 80 seconds on my system. That project has less than 100 files, that should take at most 5 seconds. Something is very off there, but it has nothing to do with fork-ts-checker-webpack-plugin

PPS: tsc --noEmit --skipLibCheck is down to 50 seconds. That's a first step.

@johnnyreilly
Copy link
Member

I'm considering a skipLibCheck tattoo. Or I would be if I wasn't already filled with regret about the noImplicitAny one I got one drunken night in 2014

@phryneas
Copy link
Contributor

phryneas commented May 3, 2019

Apart from skipLibCheck and the node_modules thing I don't see anything glaring. That project is not using much advanced type magic, so I don't understand why it could take so long. But this is definitely a TypeScript problem - maybe you could open an issue over there? A project of that size should not consume that many resources.

@phryneas
Copy link
Contributor

phryneas commented May 6, 2019

Another thing that just came up in a conversation with colleague: you should not import from @material-ui/icons directly.

Do import Close from "@material-ui/icons/Close"; instead of import {Close} from '@material-ui/icons'

maybe that helps

PS: the same thing, albeit with a lower impact applies to @material-ui/core

@phryneas
Copy link
Contributor

phryneas commented May 6, 2019

Also, you might consider downgrading to TS 3.3: microsoft/TypeScript#30663

@FishOrBear
Copy link

I just upgraded to the new version today and I have the same problem.

I tried to increase the memory, and roll back the ts version, there is no way to solve it.

  new ForkTsCheckerWebpackPlugin({ checkSyntacticErrors: true, memoryLimit: 5000 }),

@phryneas
Copy link
Contributor

Could you please specify, which packages exactly you updated and from which to which version?

  • did you only upgrade fork-ts-checker-webpack-plugin?
  • did you also upgrade typescript?

@FishOrBear
Copy link

I am testing.

When I upgraded from 0.5.2 to the latest version.
     "fork-ts-checker-webpack-plugin": "^0.5.2",

Originally it took 11s, and it became 13s after the upgrade.

@FishOrBear
Copy link

Node12
Typescript 3.3.4000
The latest fork-ts-checker-webpack-plugin

There will be no memory explosion problems.

I tried what it was caused.

@FishOrBear
Copy link

This is a magical problem, I also encountered, I tried to roll back the package, but it will also trigger the problem of insufficient memory.

And I can compile smoothly on another computer.

@FishOrBear
Copy link

tsc --noEmit
I use this script to get the same result, should it be the error caused by typescript?

@phryneas
Copy link
Contributor

phryneas commented May 13, 2019 via email

@FishOrBear
Copy link

3.3.x If properly installed, it will not cause memory leaks.

Originally I may not have correctly rolled back the typescript version.

Type checking can be done when npm i typescript@3.3.4000 is used correctly.

So I think it might be a typescript 3.4.x and 3.5.x error.

Because I run tsc --noEmit, I get the same result.

@FishOrBear
Copy link

It should be the problem of this.
microsoft/TypeScript#31227

use
npm i typescript@3.3.4000 to solve the problem

@marcusjwhelan
Copy link
Author

I tried all versions after 3.3.400 and they all had the same issue. So sticking with TS 3.3.4 for now

@edwingustafson
Copy link

Working flawlessly for me after upping the memoryLimit on the plugin itself via React App Rewired. (Because it forks, adjusting memory settings on the parent process don't help.)

config-overrides.js
https://gist.github.com/edwingustafson/ee786fe26c6a7fae394321a892698f64

Discovered my confusion by inspecting the command line via ps wwwaux on my development box: it showed the plugin's process getting the default memory limit as a command line option.

@fc
Copy link

fc commented Oct 31, 2019

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

@Chnapy
Copy link

Chnapy commented Jan 22, 2020

Hi,

I also have a memory issue on type checking, since I upgraded one of my project lib (react-scripts from 2.1 to 3.X (tried with 3.0.1, 3.2.0, 3.3.0).

Note that react-scripts uses fork-ts-checker-webpack-plugin 3.1.0 in its webpack config.
I use typescript 3.3.3 (also tried with 3.7.5).

When I try a build, it takes like 5min then fails with this error stack:

<--- Last few GCs --->

[62388:0x10280e000]   171064 ms: Scavenge 2017.5 (2093.5) -> 2002.2 (2095.0) MB, 6.5 / 0.0 ms  (average mu = 0.262, current mu = 0.223) allocation failure
[62388:0x10280e000]   171096 ms: Scavenge 2017.7 (2095.0) -> 2002.4 (2097.5) MB, 6.3 / 0.0 ms  (average mu = 0.262, current mu = 0.223) allocation failure
[62388:0x10280e000]   171124 ms: Scavenge 2017.9 (2097.5) -> 2002.6 (2099.5) MB, 7.3 / 0.0 ms  (average mu = 0.262, current mu = 0.223) allocation failure


<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x1bf3a39dbe3d]
Security context: 0x112bbff9e6e9 <JSObject>
    1: trySymbolTable(aka trySymbolTable) [0x112b05636ab1] [/Users/b046lj/Projects/afds_ext/afds-webapp/frontend/node_modules/typescript/lib/typescript.js:~33453] [pc=0x1bf3a4be076a](this=0x112bfd8826f1 <undefined>,symbols=0x112bf143fd71 <Map map = 0x112b4f684359>,ignoreQualification=0x112bfd8826f1 <undefined>)
    2: getAccessibleSymbolChainFromSymbolTable(...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10003d035 node::Abort() [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 2: 0x10003d23f node::OnFatalError(char const*, char const*) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 3: 0x1001b8e15 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 4: 0x100586d72 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 5: 0x100589845 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 6: 0x1005856ef v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 7: 0x1005838c4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 8: 0x10059015c v8::internal::Heap::AllocateRawWithLigthRetry(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
 9: 0x1005901df v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
10: 0x10055fb24 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
11: 0x1007e7e04 v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/Users/b046lj/.nvm/versions/node/v10.16.3/bin/node]
12: 0x1bf3a39dbe3d

I tried to increase the memory limit to 9000 as a test, in this case the process takes like 10min before failing (the node process takes until 9go of memory...).

There is an issue with my project types, but I have no idea where to check, the error message doesn't help me at all.

There is a way to add logs during type checking, or anything else that can help me to find the issue root cause ?
Thanks

@ahrbil
Copy link

ahrbil commented Feb 6, 2020

@Chnapy I'm having the same issue
I have turned off

"strict": false,
"strictNullChecks": false,
"strictPropertyInitialization": false,

and it's working
using typescript 3.7.5
and fork-ts-checker-webpack-plugin 4.0.3

@JosipReh
Copy link

JosipReh commented Apr 2, 2020

Did someone managed to fix this issue?

@piotr-oles
Copy link
Collaborator

Could you try fork-ts-checker-webpack-plugin@alpha? I've published a new version which potentially could resolve this issue :)

@jorisw
Copy link

jorisw commented Aug 27, 2020

I unfortunately am unable to utilize fork-ts-checker-webpack-plugin version 5.x to set the memoryLimit in my Create React App TypeScript project, because Create React App presently (3.x, 4.x-alpha) targets the fork-ts-checker-webpack-plugin 4.x configuration schema:

ValidationError: Invalid configuration object. ForkTsCheckerWebpackPlugin has been initialized using a configuration object that does not match the API schema.
 - configuration has an unknown property 'silent'. These properties are valid:
   object { async?, typescript?, eslint?, formatter?, issue?, logger? }
 - configuration.typescript should be one of these:
   boolean | object { enabled?, memoryLimit?, configFile?, context?, build?, mode?, compilerOptions?, diagnosticOptions?, extensions?, profile?, … }
   Details:
    * configuration.typescript should be a boolean.
      -> Enable TypeScript reporter.
    * configuration.typescript should be an object:
      object { enabled?, memoryLimit?, configFile?, context?, build?, mode?, compilerOptions?, diagnosticOptions?, extensions?, profile?, … }

This thread did help me however in setting the memory limit and compiling in isolation:
NODE_OPTIONS=--max-old-space-size=8192 node_modules/.bin/tsc --noEmit

... which helped me determine that I have a memory problem regardless of the memory limit: 2GB, 4GB, 8GB - same result.

So to anyone trying to override the memory limit, investigate the cause rather than to suppress the symptoms.

@seanaye
Copy link

seanaye commented Aug 30, 2020

@jorisw have you found a solution to this? Getting the same error

@jorisw
Copy link

jorisw commented Aug 31, 2020

In my case, I was importing style object definitions targeting ThemeUI, each typed as a SxStyleProp, then doing composition with the imported objects to create different styles for different states, like so:

export const someStyle: SxStyleProp {
   maxWidth: 300,
}
import {
  someStyle
} from './styles'

const manyStyles: SxStyleProp = {
  ...someStyle,
  ...(isActive ? someMoreStyles : {}),
  // ...
}

Apparently importing and composing these typed objects made the typechecker go into a memory leak of some sort. Removing the : SxStyleProp typing from each of my exported styles in styles.ts, made the problem go away.

As you can see, a pretty specific use case, so our problems are likely to be very different.

@jorisw
Copy link

jorisw commented Aug 31, 2020

Turns out the problem I was having, was related to TypeScript being bad at handling use of the spread (...) operator on objects that have union types. Using Object.assign() instead fixes it.

Discussion: system-ui/theme-ui#1162 (reply in thread)

@damienallen
Copy link

damienallen commented Nov 11, 2020

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

For anyone seeing this issue in cases where it doesn't seem related to a specific commit (i.e. roll-back doesn't work), I would like to highlight this solution. Due to some unrelated permissions issues I had temporarily made a copy of node_modules which led to this out-of-memory issue.

@radiantone
Copy link

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

For anyone seeing this issue in cases where it doesn't seem related to a specific commit (i.e. roll-back doesn't work), I would light to highlight this solution. Due to some unrelated permissions issues I had temporarily made a copy of node_modules which led to this out-of-memory issue.

Thank you for sharing this solution. It worked for me.

@gargroh
Copy link

gargroh commented Jun 11, 2021

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

Thanks @fc, this solved the problem.

@ChuckJonas
Copy link

using material ui and CRA, I run out of memory whenever I use styled components:

import Typography from '@mui/material/Typography';
import { styled } from '@mui/system';

const UnderlinedText = styled(Typography)({
    marginTop: '10px',
    color: 'rgba(0, 105, 165, 1)',
    textDecoration: 'underline',
    marginBottom: '15px',
});

export default UnderlinedText;

@piotr-oles
Copy link
Collaborator

@ChuckJonas This is most likely a TypeScript issue :)

@Mikado68
Copy link

Mikado68 commented Mar 8, 2022

Do import Close from "@material-ui/icons/Close"; instead of import {Close} from '@material-ui/icons'

Hi @phryneas

This is interesting, can you please explain (or link) the difference ?

I'm using the latter in my project to reduce the number of lines of imports, but consider changing for the former if it helps reducing the RAM consumption.

@phryneas
Copy link
Contributor

phryneas commented Mar 8, 2022

@Mikado68 this issue is 3 years old, no idea if that is still relevant. By then it was the difference between importing all icons (that file was non-treeshakable) and only one.

@okdistribute
Copy link

This helped me, thanks. I was trying to figure out why it kept crashing, and didn't think about mui. Of course!

@cpbotha
Copy link

cpbotha commented Jun 13, 2022

I had made a copy of the node_modules folder in the same directory that led to me getting this error. Deleting the copied folder resolved it for me.

Thank you very much for this @fc !

Thanks to me getting around too late to attr -s com.dropbox.ignored -V 1 node_modules aka ignoring node_modules on a new machine, it created a node_modules (Ignored Item Conflict) in the same folder. (dropbox, why can't we just add an account-global ignore-list?)

Anyways, the presence of that folder caused "issues checking in progress..." to take a minute or two and then OOM, whereas usually it takes a few seconds and does not OOM. I don't know how long it would have taken me to find the issue without @fc's comment.

How could one get tsc to output the filenames it's checking? This would probably have helped many people to find the issue.

@Twonster
Copy link

Hey guys, has anyone been able to fix the type checking memory leak issue?

@cloudever
Copy link

Seems to be the problem issuer is not that plugin or typescript itself.

For me the problem issuer was mongodb package < v5 with its declarations , upgrade to > v5 solves the problem.
You can also find issuer by paste each of your sources path (nested too) in tsconfig.js's include field, just restart app and wait for stuck. It took only 5 mins to find bad dependency in my project.

@the-rpc
Copy link

the-rpc commented Apr 21, 2023

In my case, I was importing style object definitions targeting ThemeUI, each typed as a SxStyleProp, then doing composition with the imported objects to create different styles for different states, like so:

export const someStyle: SxStyleProp {
   maxWidth: 300,
}
import {
  someStyle
} from './styles'

const manyStyles: SxStyleProp = {
  ...someStyle,
  ...(isActive ? someMoreStyles : {}),
  // ...
}

Apparently importing and composing these typed objects made the typechecker go into a memory leak of some sort. Removing the : SxStyleProp typing from each of my exported styles in styles.ts, made the problem go away.

As you can see, a pretty specific use case, so our problems are likely to be very different.

Wrestled with this issue for quite some time in a Create React App TypeScript project with MUI. Seeing you reference SxProps made me scan my project for places I was including it. Sure enough, I defined a Type that extended a MUI component and defined an optional sx property (probably wrote the Type before making it extend the MUI component). Removing the override solved the problem 😄 Thank you @jorisw

import { LoadingButtonProps } from '@mui/lab';

type ComponentProps = LoadingButtonProps & {
    sx?: SxProps; // This was the problem!
};

@Audrey-Ann
Copy link

@the-rpc we were having a similar issue with a CRA with Typescript project using MUI v6!!

You helped us pin the same issue we were experiencing when executing react-scripts build: importing SxProps directly from import { SxProps } from '@mui/system'; or even import { SxProps } from '@mui/materials/styles'; resulted in a memory leak!

The solution was to import sx's type directly from the MUI component, for instance SelectProps['sx'], ButtonProps['sx'] etc.... 😅

I hope MUI can fix the way they export their types...

@james-stuckey-lab49
Copy link

@the-rpc we were having a similar issue with a CRA with Typescript project using MUI v6!!

You helped us pin the same issue we were experiencing when executing react-scripts build: importing SxProps directly from import { SxProps } from '@mui/system'; or even import { SxProps } from '@mui/materials/styles'; resulted in a memory leak!

The solution was to import sx's type directly from the MUI component, for instance SelectProps['sx'], ButtonProps['sx'] etc.... 😅

I hope MUI can fix the way they export their types...

This fixed an issue for us on a TypeScript project using MUI as well. The memory issue is gone after removing the SxProps direct import.

@ershisan99
Copy link

Just in case it helps someone:
The issue in my case was in extending a component from our internal library using styled(Component). The component itself uses Radix under the hood, which uses some complicated types, so I assume that's what was causing the memory leak.
It appeared after bumping all the deps to latest, such as styled 5 => 6 and ts 4.x.x => 5.1.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests