Slower compilation when using @ngtools/webpack@7.1.0 #13102

filipesilva · 2018-11-30T16:49:49Z

Initially reported in angular/angular#26674 (comment) by @sod

hm indeed, just updating ngtools gives me the slowdown. Also using angular 7.1 with old ngtools 7.0.7 is significantly faster.

versions	1st run	2nd run	3rd run	4th run	5th run
`@angular 7.0.4, @ngtools 7.0.7`	31s	15s	7s	5s	5s
`@angular 7.0.4, @ngtools 7.1.0`	29s	14.5s	15.5s	12s	12s
`@angular 7.1.1, @ngtools 7.1.0`	29s	13.5s	13.5s	15s	16s
`@angular 7.1.1, @ngtools 7.0.7`	28s	12.8s	7s	5s	4.2s

The text was updated successfully, but these errors were encountered:

sod · 2018-11-30T16:56:38Z

this is how the chart looks in the 4th run with 7.0.7

and this with 7.1.0

The 7.0 log is 9MB, the 7.1 log is 75MB :D

sod · 2018-11-30T16:58:29Z

Seems like diagnostics runs even though that should run in a seperate process?

sod · 2018-11-30T17:15:16Z

I guess #12824 caused this? I'm not sure if this is the intended behavior. An opt-out would be nice.

hccampos · 2018-12-05T16:45:32Z

Just another data point here:

Angular	ngtools	1st	2nd	3rd
@angular 7.1.1	@ngtools 7.0.7	25317ms	5131ms	4316ms
@angular 7.1.1	@ngtools 7.1.0	36403ms	27959ms	27833ms

RidhwanDev · 2018-12-06T16:48:01Z

Hi, How can i downgrade @ngTools to 7.0.7?

Thanks

sod · 2018-12-06T18:15:12Z

‘npm i @ngtools/webpack@7.0‘

filipesilva · 2018-12-12T18:19:51Z

I tested this regression using https://github.com/angular/angular/tree/master/aio, rebuilding without AOT and any optimizations, and got these results:

@ngtools/webpack@7.0.7 rebuilds took ~1.25s
@ngtools/webpack@7.1.0 rebuilds took ~4.3s
@ngtools/webpack@7.1.2 rebuilds took ~1.8s (includes fix(@ngtools/webpack): avoid checking watchMode in environment hook #13133)

So I think #13133 addressed part of the problem, but there's still a regression caused by #12824.

I looked at the changes and think it's because, on JIT, syntactic diagnostics are now always gathered for the whole program. Before they were only gathered for the whole program on the first run or when the type checker was off.

But we have a code path for when less than 20 files changes that only emits those files:

angular-cli/packages/ngtools/webpack/src/angular_compiler_plugin.ts

Lines 1084 to 1108 in 603e6f4

    
           if (this._firstRun || changedTsFiles.size > 20) { 
        
             emitResult = tsProgram.emit( 
        
               undefined, 
        
               undefined, 
        
               undefined, 
        
               undefined, 
        
               { before: this._transformers }, 
        
             ); 
        
             allDiagnostics.push(...emitResult.diagnostics); 
        
           } else { 
        
             for (const changedFile of changedTsFiles) { 
        
               const sourceFile = tsProgram.getSourceFile(changedFile); 
        
               if (!sourceFile) { 
        
                 continue; 
        
               } 
        
               const timeLabel = `AngularCompilerPlugin._emit.ts+${sourceFile.fileName}+.emit`; 
        
               time(timeLabel); 
        
               emitResult = tsProgram.emit(sourceFile, undefined, undefined, undefined, 
        
                 { before: this._transformers }, 
        
               ); 
        
               allDiagnostics.push(...emitResult.diagnostics); 
        
               timeEnd(timeLabel); 
        
             } 
        
           }

In that case, we should only check diagnostics for the files we are emitting. That should reduce the rebuild time.

filipesilva · 2018-12-13T12:16:31Z

Was working a bit on this today and noticed that the results I took yesterday were of rebuilds without a vendor chunk, which great inflate total rebuild times.

To get more granular information I also used an internal profiling flag we have to see where time is spent in compilations.

The results I got were:

@ngtools/webpack@7.0.7 rebuilds took ~0.5s

i ｢wdm｣: Compiling...
TypeChecker._update: 0.173ms
AngularCompilerPlugin._createOrUpdateProgram.ts.createProgram: 8.969ms
AngularCompilerPlugin._findLazyRoutesInAst: 0.563ms
TypeChecker._createOrUpdateProgram.ts.createProgram: 14.826ms
TypeChecker.gatherDiagnostics.ts.getSyntacticDiagnostics: 0.269ms
AngularCompilerPlugin._emit.ts+D:/work/angular/aio/src/main.ts+.emit: 12.643ms
AngularCompilerPlugin._emit: 13.173ms
AngularCompilerPlugin._update._emit: 13.638ms
AngularCompilerPlugin._update: 43.407ms
AngularCompilerPlugin._make: 43.848ms
ngcLoader+D:\work\angular\aio\src\main.ts+.ngcLoader.AngularCompilerPlugin: 17.118ms
ngcLoader+D:\work\angular\aio\src\main.ts+: 20.312ms
TypeChecker.gatherDiagnostics.ts.getSemanticDiagnostics: 264.195ms
TypeChecker.message: 281.368ms

Date: 2018-12-13T11:45:50.889Z - Hash: 16ad75ffacc918bb20c2 - Time: 511ms
22 unchanged chunks
chunk {main} main.229bfd5f540aafd21011.js, main.229bfd5f540aafd21011.js.map (main) 201 kB [initial] [rendered]

@ngtools/webpack@7.1.0 rebuilds took ~3.3s

i ｢wdm｣: Compiling...
TypeChecker._update: 0.151ms
TypeChecker._createOrUpdateProgram.ts.createProgram: 10.518ms
TypeChecker.gatherDiagnostics.ts.getSemanticDiagnostics: 327.532ms
TypeChecker.message: 339.517ms
AngularCompilerPlugin._createOrUpdateProgram.ts.createProgram: 1592.257ms
AngularCompilerPlugin._emit.ts.gatherDiagnostics.ts.getSyntacticDiagnostics: 0.573ms
AngularCompilerPlugin._emit.ts+D:/work/angular/aio/src/main.ts+.emit: 197.366ms
AngularCompilerPlugin._emit: 200.312ms
AngularCompilerPlugin._update._emit: 200.506ms
AngularCompilerPlugin._update: 2803.878ms
AngularCompilerPlugin._make: 2804.271ms
ngcLoader+D:\work\angular\aio\src\main.ts+.ngcLoader.AngularCompilerPlugin: 1192.022ms
ngcLoader+D:\work\angular\aio\src\main.ts+: 1194.172ms

Date: 2018-12-13T11:50:28.263Z - Hash: 770163b2af42fff274b6 - Time: 3566ms
22 unchanged chunks
chunk {main} main.4eecb9fa84b0b62a9195.js, main.4eecb9fa84b0b62a9195.js.map (main) 201 kB [initial] [rendered]

@ngtools/webpack@7.1.2 rebuilds took ~1.1s (includes fix(@ngtools/webpack): avoid checking watchMode in environment hook #13133)

i ｢wdm｣: Compiling...
TypeChecker._update: 0.154ms
AngularCompilerPlugin._createOrUpdateProgram.ts.createProgram: 4.893ms
TypeChecker._createOrUpdateProgram.ts.createProgram: 10.950ms
TypeChecker.gatherDiagnostics.ts.getSemanticDiagnostics: 258.325ms
TypeChecker.message: 270.538ms
AngularCompilerPlugin._emit.ts.gatherDiagnostics.ts.getSyntacticDiagnostics: 0.317ms
AngularCompilerPlugin._emit.ts+D:/work/angular/aio/src/main.ts+.emit: 10.914ms
AngularCompilerPlugin._emit: 11.688ms
AngularCompilerPlugin._update._emit: 12.224ms
AngularCompilerPlugin._update: 616.467ms
AngularCompilerPlugin._make: 616.803ms
ngcLoader+D:\work\angular\aio\src\main.ts+.ngcLoader.AngularCompilerPlugin: 596.443ms
ngcLoader+D:\work\angular\aio\src\main.ts+: 598.446ms

Date: 2018-12-13T11:55:02.640Z - Hash: 9fbd23000feec6a6452b - Time: 1141ms
22 unchanged chunks
chunk {main} main.5c50f6c3037acf65f46c.js, main.5c50f6c3037acf65f46c.js.map (main) 201 kB [initial] [rendered]

@ngtools/webpack@7.1.3 rebuilds took ~1.2s (includes extra logging from refactor: add time logs in _listLazyRoutesFromProgram #13159)

i ｢wdm｣: Compiling...
TypeChecker._update: 0.151ms
AngularCompilerPlugin._createOrUpdateProgram.ts.createProgram: 5.154ms
AngularCompilerPlugin._listLazyRoutesFromProgram.createProgram: 0.065ms
TypeChecker._createOrUpdateProgram.ts.createProgram: 16.334ms
TypeChecker.gatherDiagnostics.ts.getSemanticDiagnostics: 283.161ms
TypeChecker.message: 300.672ms
AngularCompilerPlugin._listLazyRoutesFromProgram.listLazyRoutes: 630.713ms
AngularCompilerPlugin._emit.ts.gatherDiagnostics.ts.getSyntacticDiagnostics: 0.338ms
AngularCompilerPlugin._emit.ts+D:/work/angular/aio/src/main.ts+.emit: 13.726ms
AngularCompilerPlugin._emit: 14.763ms
AngularCompilerPlugin._update._emit: 15.229ms
AngularCompilerPlugin._update: 669.832ms
AngularCompilerPlugin._make: 670.866ms
ngcLoader+D:\work\angular\aio\src\main.ts+.ngcLoader.AngularCompilerPlugin: 650.003ms
ngcLoader+D:\work\angular\aio\src\main.ts+: 652.326ms

Date: 2018-12-13T12:14:41.442Z - Hash: 646a2507d4b49c72b08d - Time: 1186ms
22 unchanged chunks
chunk {main} main.68f05d7dcc3f58fc730b.js, main.68f05d7dcc3f58fc730b.js.map (main) 201 kB [initial] [rendered]

A few notes about these profiling messages:

AngularCompilerPlugin._make is the total time taken inside the ngtools/webpack plugin
TypeChecker.* refers to the forked type checker, in a separate process
Time spent in ngtools/webpack is only part of the story, as webpack also needs to recreate bundles and sourcemaps

My previous guess was that syntactic diagnostics on rebuilds were still taking too long, but that doesn't really hold true since in @ngtools/webpack@7.1.2 they are shown to take less that 1ms (AngularCompilerPlugin._emit.ts.gatherDiagnostics.ts.getSyntacticDiagnostics: 0.317ms).

What does seem to take a lot longer is AngularCompilerPlugin._update, clocking in at 43.407ms in 7.0.7 but increasing to 616.467ms in 7.1.2. The main contributor is AngularCompilerPlugin._listLazyRoutesFromProgram.listLazyRoutes, only visible in the profile logs for 7.1.3.

That increase seems related to #12418, where the way we process lazy routes and their errors changed.

djleonskennedy · 2018-12-18T14:28:24Z

Hello @filipesilva is there any news?

Wykks · 2018-12-20T09:59:59Z

I can confirm. The rebuild is slower with @ngtools/webpack 7.1.3 due to listLazyRoutes:

With

@angular-devkit/architect         0.11.3
@angular-devkit/build-angular     0.11.3
@angular-devkit/build-optimizer   0.11.3
@angular-devkit/build-webpack     0.11.3
@angular-devkit/core              7.1.3
@angular-devkit/schematics        7.1.3
@angular/cdk                      7.2.0
@angular/cli                      7.1.3
@angular/material                 7.2.0
@ngtools/webpack                  7.1.3
@schematics/angular               7.1.3
@schematics/update                0.11.3
rxjs                              6.3.3
typescript                        3.1.6
webpack                           4.23.1

cli_7_13_cpuprofile.zip

And with

@angular-devkit/architect         0.10.7
@angular-devkit/build-angular     0.10.7
@angular-devkit/build-optimizer   0.10.7
@angular-devkit/build-webpack     0.10.7
@angular-devkit/core              7.0.7
@angular-devkit/schematics        7.0.7
@angular/cdk                      7.2.0
@angular/cli                      7.0.7
@angular/compiler                 7.0.4
@angular/compiler-cli             7.0.4
@angular/material                 7.2.0
@ngtools/webpack                  7.0.7
@schematics/angular               7.0.7
@schematics/update                0.10.7
rxjs                              6.3.3
typescript                        3.1.6
webpack                           4.19.1

cli_7_07_cpuprofile.zip

clydin · 2018-12-20T15:27:09Z

The issue is indeed that within 7.1.x the Angular compiler is always used to calculate the set of lazy routes when in JIT mode. This was initially done to guarantee that the lazy routes were calculated accurately and any lazy route related errors would be shown upon a rebuild. This has the unfortunate downside of taking more time. The underlying problem is one of correctness/accuracy versus speed.

The team has decided to bring back the faster but potentially less accurate method of detecting lazy routes upon JIT rebuilds (first builds will always use the more complete Angular compiler method). Applications that do not have lazy routes within libraries and that only use direct string literals with loadChildren should not be affected by the potential of less accurate detection. Note that the function overload of loadChildren also does not apply to this situation.
For those projects where correctness of lazy route detection outweighs rebuild speed, please consider using AOT mode for development. AOT mode will also provide a full set of template errors as well which JIT mode is not capable of doing.

If you have concerns about this decision, please let us know.

why520crazy · 2018-12-27T06:06:36Z

When can we fix this problem？I'm looking forward to resolve it. Thanks!

why520crazy · 2018-12-27T06:14:35Z

@filipesilva In my case, run npm run build:demo takes 30 seconds use @angular/cli 6.1.5.

"@angular-devkit/build-angular": "~0.6.8",
"@angular-devkit/build-ng-packagr": "^0.7.5",
"@angular/cli": "~6.1.5",
"@angular/compiler-cli": "^6.1.6",

Once I upgrade @angular/cli to 7.1.4 , it takes 88 seconds.

my repo is https://github.com/worktile/ngx-tethys, you can reproduce use npm run build:demo

alan-agius4 · 2018-12-27T10:20:11Z

@why520crazy, I'll be working on this today.

clydin · 2018-12-27T16:50:17Z

@why520crazy this issue is regarding rebuild performance in JIT mode and will have no effect on initial builds or production (AOT) builds.

Your issue is more likely due to this terser (the JavaScript minifier) issue: terser/terser#204

why520crazy · 2018-12-28T01:56:11Z

@clydin I see, thanks.
I think both should be needed.

jochenjonc · 2019-01-02T15:47:29Z

@why520crazy this issue is regarding rebuild performance in JIT mode and will have no effect on initial builds or production (AOT) builds.

Your issue is more likely due to this terser (the JavaScript minifier) issue: terser-js/terser#204

I had the same issue and forcing a higher version of terser fixed my build.

npm install terser@latest --save-dev

This reverts commit edb84b3 The team has decided to bring back the faster but potentially less accurate method of detecting lazy routes upon JIT rebuilds (first builds will always use the more complete Angular compiler method). Applications that do not have lazy routes within libraries and that only use direct string literals with loadChildren should not be affected by the potential of less accurate detection. Note that the function overload of loadChildren also does not apply to this situation. For those projects where correctness of lazy route detection outweighs rebuild speed, please consider using AOT mode for development. AOT mode will also provide a full set of template errors as well which JIT mode is not capable of doing. Fixes #13102

HarelM · 2019-01-11T22:41:57Z

I'm using 7.2.1 and it seems like the problem still exists - I'm not getting the incremental build times I was used to - I got used to changing the code and refreshing the browser to see the changes.
Now I change the code, wait for a few seconds to let the cli finish and only the refresh the browser... :-(

Date: 2019-01-11T22:38:01.142Z
Hash: ad9c4eaf6e8a88691976
Time: 39639ms
chunk {main} main.js, main.js.map (main) 2.15 MB [initial] [rendered]
chunk {polyfills} polyfills.js, polyfills.js.map (polyfills) 408 kB [initial] [rendered]
chunk {runtime} runtime.js, runtime.js.map (runtime) 6.08 kB [entry] [rendered]
chunk {scripts} scripts.js, scripts.js.map (scripts) 470 bytes  [rendered]
chunk {styles} styles.js, styles.js.map (styles) 694 kB [initial] [rendered]
chunk {vendor} vendor.js, vendor.js.map (vendor) 13.1 MB [initial] [rendered]

Date: 2019-01-11T22:38:40.799Z - Hash: ad9c4eaf6e8a88691976 - Time: 25970ms
6 unchanged chunks

Date: 2019-01-11T22:39:06.759Z - Hash: ad9c4eaf6e8a88691976 - Time: 10690ms
6 unchanged chunks

Date: 2019-01-11T22:39:22.465Z - Hash: ad9c4eaf6e8a88691976 - Time: 9845ms
6 unchanged chunks

Date: 2019-01-11T22:39:53.982Z - Hash: ad9c4eaf6e8a88691976 - Time: 18699ms
6 unchanged chunks

psurrey · 2019-01-16T17:31:00Z

I see that this has been fixed by a revert. Any plans when this is going to be released?

clydin · 2019-01-16T17:37:43Z

@psurrey It already has been. Ensure that the project is using both @angular/cli@7.2.1 and @angular-devkit/build-angular@0.12.1.

HarelM · 2019-01-16T19:45:53Z

@clydin thanks for pointing out that build-angular update is also needed!
Rebuild times have been restored to a reasonable amount of time.

angular-automatic-lock-bot · 2019-09-09T04:41:55Z

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}

This reverts commit edb84b3 The team has decided to bring back the faster but potentially less accurate method of detecting lazy routes upon JIT rebuilds (first builds will always use the more complete Angular compiler method). Applications that do not have lazy routes within libraries and that only use direct string literals with loadChildren should not be affected by the potential of less accurate detection. Note that the function overload of loadChildren also does not apply to this situation. For those projects where correctness of lazy route detection outweighs rebuild speed, please consider using AOT mode for development. AOT mode will also provide a full set of template errors as well which JIT mode is not capable of doing. Fixes angular#13102

filipesilva added type: bug/fix freq3: high severity5: regression area: ngtools/webpack labels Nov 30, 2018

ngbot bot added this to the Backlog milestone Nov 30, 2018

filipesilva mentioned this issue Nov 30, 2018

@ngtools/webpack / ngc - 4x slower aot compilation from ng6 to ng7 angular/angular#26674

Closed

This was referenced Dec 2, 2018

Debug watch builds are slower in angular 7 #13112

Closed

Extremely slow ng serve performance [Source Map compiling] #13087

Closed

clydin mentioned this issue Dec 5, 2018

fix(@ngtools/webpack): avoid checking watchMode in environment hook #13133

Merged

alexeagle closed this as completed in #13133 Dec 6, 2018

filipesilva reopened this Dec 12, 2018

filipesilva added the needs: discussion On the agenda for team meeting to determine next steps label Dec 13, 2018

kondi mentioned this issue Dec 17, 2018

v7.rc.0 serving 2x - 3x slower than stable versions on my projects #12432

Closed

hansl removed the needs: discussion On the agenda for team meeting to determine next steps label Dec 20, 2018

alan-agius4 mentioned this issue Dec 27, 2018

revert: fix(@ngtools/webpack): emit lazy routes errors on rebuilds #13309

Merged

filipesilva mentioned this issue Dec 27, 2018

v7.0.0 bugs and 4/5x slower serve & compile AOT #12646

Closed

filipesilva pinned this issue Dec 27, 2018

alan-agius4 unpinned this issue Dec 27, 2018

filipesilva mentioned this issue Dec 27, 2018

Build with "ng -prod" is extremely slow #6795

Closed

alan-agius4 pinned this issue Dec 27, 2018

mgechev unpinned this issue Jan 4, 2019

alexeagle closed this as completed in #13309 Jan 8, 2019

angular-automatic-lock-bot bot locked and limited conversation to collaborators Sep 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Slower compilation when using @ngtools/webpack@7.1.0 #13102

Slower compilation when using @ngtools/webpack@7.1.0 #13102

filipesilva commented Nov 30, 2018

sod commented Nov 30, 2018 •

edited

sod commented Nov 30, 2018

sod commented Nov 30, 2018 •

edited

hccampos commented Dec 5, 2018

RidhwanDev commented Dec 6, 2018

sod commented Dec 6, 2018

filipesilva commented Dec 12, 2018

filipesilva commented Dec 13, 2018

djleonskennedy commented Dec 18, 2018

Wykks commented Dec 20, 2018 •

edited

clydin commented Dec 20, 2018 •

edited

why520crazy commented Dec 27, 2018

why520crazy commented Dec 27, 2018 •

edited

alan-agius4 commented Dec 27, 2018

clydin commented Dec 27, 2018

why520crazy commented Dec 28, 2018

jochenjonc commented Jan 2, 2019

HarelM commented Jan 11, 2019

psurrey commented Jan 16, 2019

clydin commented Jan 16, 2019

HarelM commented Jan 16, 2019

angular-automatic-lock-bot bot commented Sep 9, 2019

Slower compilation when using @ngtools/webpack@7.1.0 #13102

Slower compilation when using @ngtools/webpack@7.1.0 #13102

Comments

filipesilva commented Nov 30, 2018

sod commented Nov 30, 2018 • edited

sod commented Nov 30, 2018

sod commented Nov 30, 2018 • edited

hccampos commented Dec 5, 2018

RidhwanDev commented Dec 6, 2018

sod commented Dec 6, 2018

filipesilva commented Dec 12, 2018

filipesilva commented Dec 13, 2018

djleonskennedy commented Dec 18, 2018

Wykks commented Dec 20, 2018 • edited

clydin commented Dec 20, 2018 • edited

why520crazy commented Dec 27, 2018

why520crazy commented Dec 27, 2018 • edited

alan-agius4 commented Dec 27, 2018

clydin commented Dec 27, 2018

why520crazy commented Dec 28, 2018

jochenjonc commented Jan 2, 2019

HarelM commented Jan 11, 2019

psurrey commented Jan 16, 2019

clydin commented Jan 16, 2019

HarelM commented Jan 16, 2019

angular-automatic-lock-bot bot commented Sep 9, 2019

sod commented Nov 30, 2018 •

edited

sod commented Nov 30, 2018 •

edited

Wykks commented Dec 20, 2018 •

edited

clydin commented Dec 20, 2018 •

edited

why520crazy commented Dec 27, 2018 •

edited