v7.0.0 bugs and 4/5x slower serve & compile AOT

[elvisbegovic]

I hesitate to open this issue on angular page. I opened same issue title 2 week ago here without AOT : v7.rc.0 serving 2x - 3x slower than stable versions on my projects after moving to angular 7.0.0 all is slower (jit seems OK)

I'm submitting a...

[ x ] Performance issue
[ x ] Bugs

Current behavior

All was OK with cli7rc3 and angular 6.1.10
This morning after updating to angular 7.0.0 serving with JIT is ok but serving with AOT or build my application or generating i18n xmb file is 2-3 times slower

Expected behavior

Get build/extract i18n time like with cli7rc3 and angular 6.1.10

1. Minimal info of AOT serving

0. My project have >1200 files.
1. Before update node --max-old-space-size=8192 ./node_modules/@angular/cli/bin/ng serve --aot --sourceMap=false --disableHostCheck --port=4200 first build finish in 45seconds and Now blocking:

3mn => 11% building modules 9/10 modules 1 active ...ojects\i10\src\assets\styles\styles.css and rest steps seems ok It finish in 228737ms

2. Before update refresh(file edit) it finish in 30seconds and 3th refresh in 25seconds and Now it refresh in 200277ms and 3th refresh in WARNING : 381979ms it is >6 minutes

2. Minimal info of extract i18n

1. Extract is 3-4 times slower than before update

Here is my angular.json config file

I tried debug with ndb and share my profile data with ng s --aot but it seems not work ndb crash ( disk is highly used...):

Environment

Angular CLI: 7.0.1
Node: 10.12.0
OS: win32 x64
Angular: 7.0.0
... common, compiler, compiler-cli, core, forms, http
... language-service, platform-browser, platform-browser-dynamic
... router

Package                           Version
-----------------------------------------------------------
@angular-devkit/architect         0.10.1
@angular-devkit/build-angular     0.9.0-rc.3
@angular-devkit/build-optimizer   0.9.0-rc.3
@angular-devkit/build-webpack     0.9.0-rc.3
@angular-devkit/core              7.0.1
@angular-devkit/schematics        7.0.1
@angular/cdk                      6.4.7
@angular/cli                      7.0.1
@angular/material                 6.4.7
@ngtools/webpack                  7.0.0-rc.3
@schematics/angular               7.0.1
@schematics/update                0.10.1
rxjs                              6.3.3
typescript                        3.1.3
webpack                           4.19.1

Others:
maybe related to : angular/angular#26572 , #12645

@filipesilva heya, sorry to make noise again if you consider this as angular compiler problem(as I think) close or move it please

[filipesilva]

@istiti did the size of you main bundle increase as well after the update? I've read a report that the angular compiler was being included in bundles which can cause longer builds (this is unintended).

Also cc @ocombe about the i18n error.

[filipesilva]

@istiti can you try @@angular-devkit/build-angular@~0.10.0 and tell me if the AOT rebuild times are still very long?

[hmagrini]

@filipesilva not OP, but I'm seeing similar long times with @angular-devkit/build-angular@~0.10.1 as well. Main bundle size increased by ~20%.

[elvisbegovic]

@filipesilva main bundle maybe about +0.3mb but maybe this is becasue code size changed... i can't really say it is bigger I find it is not increasing!

right I was with build-angular on 0.9.rc3 after updating using ng udpate --next and didn't realize this build-angular didn't automatically update so after move on 0.10.1 nothing change... same slowdown!

I remove package-lock, remove nodes_modules folder, npm i again on this package file

[filipesilva]

EDIT: that size difference between this repro and a new project seemed to be the router, so it didn't actually show a regression.

Original comment below.

---

@crisbeto provided a reproduction (https://github.com/filipesilva/cli-seven-size-regression) where the size regression can be seen:

$ ng build --prod

Date: 2018-10-19T10:23:51.856Z
Hash: c0ef836735b53b519186
Time: 31168ms
chunk {0} runtime.ec2944dd8b20ec099bf3.js (runtime) 1.44 kB [entry] [rendered]
chunk {1} main.9c9c99a7a2fa8cb359cc.js (main) 259 kB [initial] [rendered]
chunk {2} polyfills.c6871e56cb80756a5498.js (polyfills) 38.4 kB [initial] [rendered]
chunk {3} styles.3bb2a9d4949b7dc120a9.css (styles) 0 bytes [initial] [rendered]

Notice the main bundle is 259kB.

I tried to make a new project and actually couldn't see the regression, the size was pretty much the same as in v6 (170 vs 173kb):

$ ng build --prod

Date: 2018-10-19T10:11:55.402Z
Hash: ec058cf8c6e7c3d3766f
Time: 25655ms
chunk {0} runtime.ec2944dd8b20ec099bf3.js (runtime) 1.44 kB [entry] [rendered]
chunk {1} main.473563a0e53e983e2153.js (main) 173 kB [initial] [rendered]
chunk {2} polyfills.c6871e56cb80756a5498.js (polyfills) 38.4 kB [initial] [rendered]
chunk {3} styles.3bb2a9d4949b7dc120a9.css (styles) 0 bytes [initial] [rendered]

But since I have a repro I can follow up.

[elvisbegovic]

I confirm On new green project all seems okay @filipesilva

[dottodot]

My main bundle has gone from 1.33MB to 1.59 MB which makes v7 unuseable, 1.3Mb was already on the limit of what's acceptable and can't really afford it to be any bigger. Where has the extra 0.26MB come from?

[skreborn]

@filipesilva The difference between 173 kB and 259 kB is simply initializing the latter with the router module included. Started two fresh projects to confirm.

[skreborn]

Including the router module apparently adds an extra ~3400 (prettified) lines to the main bundle.

[filipesilva]

@dottodot I don't know where those extra 0.26 MB come from but I think they hold the answer to this problem.

Would you be willing to run some tests for me?

The first thing to figure out is if the increased size came from updating Angular itself or the CLI. We can figure that out by updating them separately.

By "Angular itself" I mean all @angular/* dependencies except @angular/cli. For v6 you can use 6.1.7 and for v7 you can use 7.0.0.

By CLI I mean @angular/cli and @angular-devkit/build-angular. For v6 you can use @angular/cli@6.2.5 and @angular-devkit/build-angular@0.8.5. For v7 you can use @angular/cli@7.0.1 and @angular-devkit/build-angular@0.10.1.

I assume you're doing a production build. Can you tell me the size of the main bundle for these combinations of dependencies please?

• Angular v6 and CLI v6
• Angular v6 and CLI v7
• Angular v7 and CLI v7

This should help narrow down where the size changed.

[filipesilva]

@skreborn yeah it took me a while to figure that out, I've edited the original comment to reflect it :/

The size difference must be something else.

[dottodot]

@filipesilva no problem will try and get that info together will be a bit later on though.

[dottodot]

@filipesilva

Angular v6 and CLI v6 1.36 MB
Angular v6 and CLI v7 1.36 MB
Angular v7 and CLI v7 1.59 MB

So I'm assuming that means it's an angular issue rather than cli.

[skreborn]

@dottodot Is the increase in build time similarly noticeable between those versions?

[filipesilva]

@crisbeto also reports it happens when updating Angular v6 to v7, while keeping CLI at v7.

I'm trying to get to the bottom of it but it's been really hard what a reproduction. If anyone has a project where they see this and can share, it would really help.

[coryrylan]

A private project I tried to upgrade did increase in bundle size due to the compiler it looks like being bundled into the production output. I am hoping to try to recreate this is in a isolated small repo. The project is fairly large 60,000+ lines of TS. Here are the images of the webpack bundle analyzer before upgrading to 7 and after.

[aalbericio]

Thanks for your clarification, Filipe.

Now that we know that this is an expected behavior, I can now be confident in increasing that memory setting.

I didn't want to do that before since I always expect new versions to be faster, better and use less memory :)

[kzc]

To be honest I expected terser to require less memory than uglify-es, because of existing bugs that were fixed.

@filipesilva No memory improvement was made in terser over uglify-es. Perhaps in the course of fixing bugs the memory requirement may have actually increased in some cases. If you manage to isolate a terser minify() JS input that has significantly larger memory requirements than uglify-es with the same minify options, put the JS in question into a gist and create a new terser issue.

The one dramatic improvement for the minify time regression in collapse_vars was capping the upper bound of quadratic complexity to save CPU time - not memory.

what can happen is that the inner module closures (the module concatenation thing I mentioned before) managed to get more code in a single module and thus require more memory to optimize, with potential better savings

That's probably what's going on. More code was likely fed to terser in a more recent version of webpack or angular-cli. Perhaps there are webpack dials to limit the size of chunks.

[kzc]

Also keep in mind that one must take into account the aggregate memory consumed by angular-cli, webpack and the minifier. Because the minifier is the last to run it often is blamed for problems occurring in earlier phases of the build pipeline. Webpack had several memory problems related to chunks in the past as I recall.

[filipesilva]

@kzc thanks for chiming in, it's always nice to have your uglify insight!

I kinda assumed that the memory usage came hand in hand with the CPU consumption since as one gets closer to the memory limit more and more time is spent on garbage collection. But I don't really have the logs of any particular project... source files for these bigger projects is hard to come by.

I had a report in #11996 of a 60% reduction in build time when moving to terser though. Maybe that was just the regression you mentioned.

Anyway if I come across specific input that exhibits odd resource consumption I'll file an issue.

In regards to the aggregate memory usage, we do use the parallel processing in https://github.com/webpack-contrib/terser-webpack-plugin#parallel. This ends up running individual chunks through terser in a separate process, so the overhead from other tooling is reduced.

However the chunks are rarely of similar size. Most of the time there's a single huge chunk and several small ones. I think the huge chunk is the one that busts the memory limit.

Now that I think about it, I should check if the parent processes memory limit is being passed to the worker processes there...

[kzc]

Here's an example of a webpack process being very close to the Node heap threshold and a webpack + minifier upgrade just put it over the limit: webpack-contrib/uglifyjs-webpack-plugin#188

I kinda assumed that the memory usage came hand in hand with the CPU consumption since as one gets closer to the memory limit more and more time is spent on garbage collection.

True, CPU and memory use are often correlated but minifiers (and probably most optimizing compilers in general) spend the majority of their time looking for optimization opportunities in the AST that may or may not exist. These AST searches don't usually create much in the way of long lived heap objects.

Recent V8 versions reportedly have better GC algorithms, but there also could be regressions in some cases. So it would be worth testing different Node versions when encountering a memory issue.

[volago]

In my case everything was fine after ng update to v7. But once I updated devDependencies:

the production build time increased about 3 times:

[filipesilva]

@volago can you try the latest @angular-devkit/build-angular@0.10.6 instead please? The version you are using is old.

[volago]

@filipesilva I've tried with 0.6.8 because it's marked as LTS. Latest (0.10.6) as well as next (0.11.0-rc.0 ) also slows down the build.

[filipesilva]

@volago is your project somewhere I can look at?

[volago]

@filipesilva is't private, sorry.

[Hypenate]

Just hopping in on this.
I was getting an error (call_and_retry_last allocation) when I would ng build --prod my project, ng serve and ng build worked fine.
The ng build --prod would run forever until it threw this error.

I updated to the latest node.js and turned sourceMap: true to sourceMap: false in both tsconfig.js and angular.json.

[inthegarage]

Instead of the number 8000 (max_old_space_size) being bandied about, our medium sized project needed a value of 16000 to get going. This 1/2ed the time taken for the build.

[fxck]

Ever since updating to

"@angular-devkit/build-angular": "~0.11.0",
"@angular/cli": "7.1.0"

serve/jit rebuild times got like ten times slower..

(running it with "start:jit": "node --max_old_space_size=16384 ./node_modules/@angular/cli/bin/ng serve")

[jtyagi]

@filipesilva getting ERROR in Internal Error: The structure of the program changed during codegen..
while creating build using ng build --prod. it is working fine for ng build or ng serve.
while upgrading my application to angular 7.
"@angular/cli": "^7.1.0",
"@angular/compiler-cli": "~7.1.0"
"@angular/core": "~7.1.0"

angular.json configuration
"production": {
"optimization": true,
"outputHashing": "all",
"sourceMap": false,
"extractCss": true,
"namedChunks": false,
"aot": true,
"extractLicenses": false,
"vendorChunk": false,
"buildOptimizer": true,
"fileReplacements": [
{
"replace": "src/environments/environment.ts",
"with": "src/environments/environment.prod.ts"
}
]
}

[titov-dmytro]

Same here

[HarelM]

Just upgraded from 6.0 to 7.1 and build times of both production and incremental became unreasonable.
I have opened a separate issue here: #13112 which contains all the relevant data.
It seems that I'm now also getting javascript heap as described here: #5618.
Can be seen in latest appveyor build:
https://ci.appveyor.com/project/IsraelHikingHost/site
Should I downgrade back to 6? Does anyone in this thread have an experience with a version that is working as expected?

[no-more]

Is it not possible to enable cache like cache-loader? Maybe it have already been asked and answered but when I was using webpack + ts-loader + cache-loader + thread-loader I was able to reduce my build time by 3-4 and rebuild time by almost 10. Watch was also greatly improved.

I had to switch to angular-cli to benefit AOT witch is not an option (JIT was taking over 30s on an high end computer).
I really like angular cli and schematics. But build, rebuild and watch times get me crazy.

If it was possible to improve build time with webpack it should be possible to do so with angular-cli, no ?

Thanks

[parliament718]

@HarelM cli v 7.1 was unbearably slow for me. Reverting to 6.2.8 gave me a drastic improvement

[aalbericio]

I'd like to ask a little question regarding the future...

What is the expected impact of Webpack 5 in this issue? Is there any numbers Google can share on this?

Thanks

[matheusdavidson]

Also have slower live reloading with angular 7. Was way quicker with angular 6.

[Meligy]

Has anyone checked if the situation is different with Bazel running the compilation?

As explained by @mgechev in his post Introducing Bazel Schematics for Angular CLI.

[filipesilva]

Just wanted to mention there was a regression in 7.1 (#13102), and we have a fix on the way for it.

[StuartMc77]

Has anyone got a definite fix for this yet? I created a new Angular 7 project (version details below) and running any ng commands while in the app folder is taking a ridiculous about of time, for example, running "ng --version" outside the app folder takes about 5 seconds, running it INSIDE the app folder took around 10 minutes.

VERSION DETAILS (ng --version):
Angular CLI: 7.2.1
Node: 8.11.3
OS: win32 x64
Angular: 7.2.0
... animations, common, compiler, compiler-cli, core, forms
... language-service, platform-browser, platform-browser-dynamic
... router

Package Version

@angular-devkit/architect 0.12.1
@angular-devkit/build-angular 0.12.1
@angular-devkit/build-optimizer 0.12.1
@angular-devkit/build-webpack 0.12.1
@angular-devkit/core 7.2.1
@angular-devkit/schematics 7.2.1
@angular/cli 7.2.1
@ngtools/webpack 7.2.1
@schematics/angular 7.2.1
@schematics/update 0.12.1
rxjs 6.3.3
typescript 3.2.2
webpack 4.23.1

UPDATE
Through trial and error I have discovered my problem is something to do with the IDE I am using, VS Code. When it is closed I can run any ng command, i.e. --version or serve, and it runs as quickly as I expect. When VS Code is open, AND my angular apps folder is opened in it, it slows down again.

Currently debugging trying to pinpoint what VS Code is doing to slow it down.

[subodhkumar]

To be honest I expected terser to require less memory than uglify-es, because of existing bugs that were fixed.

@filipesilva No memory improvement was made in terser over uglify-es. Perhaps in the course of fixing bugs the memory requirement may have actually increased in some cases. If you manage to isolate a terser minify() JS input that has significantly larger memory requirements than uglify-es with the same minify options, put the JS in question into a gist and create a new terser issue.

The one dramatic improvement for the minify time regression in collapse_vars was capping the upper bound of quadratic complexity to save CPU time - not memory.

what can happen is that the inner module closures (the module concatenation thing I mentioned before) managed to get more code in a single module and thus require more memory to optimize, with potential better savings

That's probably what's going on. More code was likely fed to terser in a more recent version of webpack or angular-cli. Perhaps there are webpack dials to limit the size of chunks.

@filipesilva & @kzc Thanks for providing details on this.

Just wanted to know where can we find the webpack dials to check the chunk size.
Shouldnt the size of chunks be limited not to cross node limit, as it cause errors(I agree on the performance reason though).. a more info on the same would help.

Thanks again!

[mp54214]

With Angular 8 I had the same issue. I updated "@angular-devkit/build-angular": "^0.800.6"
to latest "@angular-devkit/build-angular": "^0.801.0" and ng serve build / rebuild times dropped down to what I could call normal (i.e. very quick).

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}