Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[experiment] Run babel over output for let/const to var #52656

Draft
wants to merge 15 commits into
base: main
Choose a base branch
from
Draft

[experiment] Run babel over output for let/const to var #52656

wants to merge 15 commits into from

Conversation

jakebailey
Copy link
Member

@jakebailey jakebailey commented Feb 7, 2023 •
edited

This is a performance boost we already knew we'd be losing when switching to modules + esbuild, as the latter doesn't support this downleveling (yet?).

I want to retest to see how good this would be, it's probably going to be worth it to do some form of this.

@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Feb 7, 2023 •
edited

Heya @jakebailey, I've started to run the perf test suite on this PR at 1a42515. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - main..52656
Metric main 52656 Delta Best Worst p-value
Angular - node (v18.10.0, x64)
Memory used 359,797k (± 0.02%) 358,447k (± 0.03%) -1,350k (- 0.38%) 358,300k 358,548k p=0.005 n=6
Parse Time 3.93s (± 0.90%) 3.52s (± 0.45%) 🟩-0.41s (-10.44%) 3.50s 3.54s p=0.005 n=6
Bind Time 1.17s (± 0.64%) 1.13s (± 0.46%) 🟩-0.04s (- 3.27%) 1.13s 1.14s p=0.004 n=6
Check Time 8.71s (± 0.72%) 8.46s (± 0.43%) -0.25s (- 2.93%) 8.42s 8.52s p=0.005 n=6
Emit Time 7.61s (± 0.87%) 7.39s (± 1.32%) -0.22s (- 2.91%) 7.26s 7.55s p=0.008 n=6
Total Time 21.42s (± 0.68%) 20.50s (± 0.67%) 🟩-0.92s (- 4.29%) 20.34s 20.73s p=0.005 n=6
Compiler-Unions - node (v18.10.0, x64)
Memory used 193,220k (± 1.19%) 194,214k (± 1.58%) ~ 191,380k 197,145k p=0.689 n=6
Parse Time 1.63s (± 1.32%) 1.48s (± 0.35%) 🟩-0.15s (- 9.28%) 1.48s 1.49s p=0.004 n=6
Bind Time 0.80s (± 0.65%) 0.77s (± 0.67%) 🟩-0.03s (- 3.77%) 0.76s 0.77s p=0.004 n=6
Check Time 9.75s (± 0.52%) 9.56s (± 0.41%) -0.19s (- 1.97%) 9.51s 9.61s p=0.005 n=6
Emit Time 2.80s (± 1.01%) 2.73s (± 1.05%) -0.07s (- 2.38%) 2.69s 2.77s p=0.008 n=6
Total Time 14.98s (± 0.46%) 14.54s (± 0.40%) -0.44s (- 2.90%) 14.48s 14.61s p=0.005 n=6
Monaco - node (v18.10.0, x64)
Memory used 343,965k (± 0.01%) 342,807k (± 0.01%) -1,159k (- 0.34%) 342,750k 342,834k p=0.005 n=6
Parse Time 2.94s (± 0.84%) 2.60s (± 1.13%) 🟩-0.33s (-11.40%) 2.55s 2.63s p=0.005 n=6
Bind Time 1.04s (± 1.32%) 1.02s (± 1.01%) ~ 1.01s 1.04s p=0.098 n=6
Check Time 7.07s (± 0.47%) 6.89s (± 0.35%) -0.18s (- 2.57%) 6.86s 6.91s p=0.005 n=6
Emit Time 4.33s (± 1.22%) 4.21s (± 0.32%) -0.12s (- 2.73%) 4.19s 4.23s p=0.005 n=6
Total Time 15.37s (± 0.45%) 14.72s (± 0.33%) 🟩-0.64s (- 4.19%) 14.68s 14.79s p=0.005 n=6
TFS - node (v18.10.0, x64)
Memory used 300,377k (± 0.01%) 299,321k (± 0.01%) -1,055k (- 0.35%) 299,297k 299,343k p=0.005 n=6
Parse Time 2.27s (± 1.71%) 2.10s (± 1.11%) 🟩-0.18s (- 7.77%) 2.07s 2.13s p=0.005 n=6
Bind Time 1.17s (± 0.44%) 1.14s (± 0.45%) -0.03s (- 2.57%) 1.13s 1.14s p=0.004 n=6
Check Time 6.54s (± 0.65%) 6.47s (± 0.62%) -0.08s (- 1.20%) 6.41s 6.51s p=0.020 n=6
Emit Time 3.93s (± 1.01%) 3.82s (± 0.66%) -0.11s (- 2.76%) 3.79s 3.85s p=0.005 n=6
Total Time 13.92s (± 0.51%) 13.52s (± 0.35%) -0.40s (- 2.85%) 13.45s 13.57s p=0.005 n=6
material-ui - node (v18.10.0, x64)
Memory used 476,782k (± 0.01%) 475,737k (± 0.01%) -1,045k (- 0.22%) 475,696k 475,792k p=0.005 n=6
Parse Time 3.53s (± 0.46%) 3.11s (± 0.24%) 🟩-0.42s (-11.77%) 3.10s 3.12s p=0.005 n=6
Bind Time 0.97s (± 0.53%) 0.93s (± 0.87%) 🟩-0.03s (- 3.45%) 0.92s 0.94s p=0.004 n=6
Check Time 17.12s (± 0.55%) 17.06s (± 0.63%) ~ 16.89s 17.21s p=0.469 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.61s (± 0.38%) 21.11s (± 0.52%) -0.51s (- 2.35%) 20.93s 21.25s p=0.005 n=6
xstate - node (v18.10.0, x64)
Memory used 549,092k (± 0.03%) 546,665k (± 0.02%) -2,426k (- 0.44%) 546,562k 546,807k p=0.005 n=6
Parse Time 4.58s (± 0.67%) 4.03s (± 0.36%) 🟩-0.55s (-12.07%) 4.01s 4.05s p=0.005 n=6
Bind Time 1.71s (± 1.26%) 1.67s (± 0.82%) -0.05s (- 2.82%) 1.65s 1.69s p=0.006 n=6
Check Time 2.81s (± 0.54%) 2.75s (± 0.92%) -0.06s (- 2.31%) 2.73s 2.80s p=0.007 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 6.19%) ~ 0.08s 0.09s p=0.174 n=6
Total Time 9.20s (± 0.25%) 8.53s (± 0.38%) 🟩-0.67s (- 7.32%) 8.49s 8.59s p=0.005 n=6
Angular - node (v16.17.1, x64)
Memory used 359,144k (± 0.01%) 357,763k (± 0.01%) -1,381k (- 0.38%) 357,733k 357,806k p=0.005 n=6
Parse Time 4.17s (± 0.45%) 3.69s (± 0.44%) 🟩-0.47s (-11.32%) 3.68s 3.72s p=0.005 n=6
Bind Time 1.24s (± 0.41%) 1.19s (± 0.86%) 🟩-0.05s (- 4.02%) 1.18s 1.21s p=0.004 n=6
Check Time 9.49s (± 0.19%) 9.23s (± 0.23%) -0.26s (- 2.72%) 9.21s 9.27s p=0.005 n=6
Emit Time 8.06s (± 0.35%) 7.91s (± 0.73%) -0.15s (- 1.90%) 7.84s 7.99s p=0.004 n=6
Total Time 22.96s (± 0.23%) 22.02s (± 0.32%) 🟩-0.94s (- 4.07%) 21.96s 22.14s p=0.005 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,814k (± 0.04%) 193,031k (± 0.02%) -783k (- 0.40%) 192,959k 193,064k p=0.005 n=6
Parse Time 1.80s (± 0.93%) 1.57s (± 0.87%) 🟩-0.23s (-12.96%) 1.55s 1.59s p=0.005 n=6
Bind Time 0.84s (± 0.00%) 0.82s (± 0.50%) -0.02s (- 2.18%) 0.82s 0.83s p=0.002 n=6
Check Time 10.42s (± 0.35%) 10.10s (± 0.73%) 🟩-0.32s (- 3.04%) 10.01s 10.20s p=0.005 n=6
Emit Time 3.10s (± 4.70%) 3.01s (± 0.91%) 🟩-0.10s (- 3.17%) 2.97s 3.05s p=0.024 n=6
Total Time 16.17s (± 1.07%) 15.50s (± 0.38%) 🟩-0.67s (- 4.16%) 15.41s 15.55s p=0.005 n=6
Monaco - node (v16.17.1, x64)
Memory used 343,232k (± 0.01%) 342,073k (± 0.01%) -1,159k (- 0.34%) 342,035k 342,106k p=0.005 n=6
Parse Time 3.14s (± 1.50%) 2.78s (± 1.13%) 🟩-0.36s (-11.52%) 2.74s 2.82s p=0.005 n=6
Bind Time 1.11s (± 1.51%) 1.08s (± 0.48%) -0.03s (- 2.40%) 1.08s 1.09s p=0.024 n=6
Check Time 7.73s (± 0.25%) 7.50s (± 0.39%) 🟩-0.23s (- 3.02%) 7.46s 7.53s p=0.005 n=6
Emit Time 4.54s (± 0.46%) 4.46s (± 0.94%) -0.08s (- 1.73%) 4.40s 4.51s p=0.005 n=6
Total Time 16.51s (± 0.50%) 15.82s (± 0.50%) 🟩-0.70s (- 4.22%) 15.69s 15.90s p=0.005 n=6
TFS - node (v16.17.1, x64)
Memory used 299,706k (± 0.01%) 298,645k (± 0.01%) -1,061k (- 0.35%) 298,626k 298,663k p=0.005 n=6
Parse Time 2.46s (± 0.99%) 2.21s (± 0.50%) 🟩-0.25s (-10.16%) 2.20s 2.22s p=0.004 n=6
Bind Time 1.26s (± 0.82%) 1.23s (± 1.03%) -0.03s (- 2.12%) 1.22s 1.25s p=0.012 n=6
Check Time 7.21s (± 0.39%) 7.09s (± 0.61%) -0.12s (- 1.66%) 7.05s 7.15s p=0.005 n=6
Emit Time 4.22s (± 0.21%) 4.31s (± 0.76%) +0.09s (+ 2.05%) 4.27s 4.35s p=0.005 n=6
Total Time 15.15s (± 0.23%) 14.84s (± 0.46%) -0.31s (- 2.04%) 14.76s 14.94s p=0.005 n=6
material-ui - node (v16.17.1, x64)
Memory used 476,012k (± 0.01%) 475,060k (± 0.01%) -952k (- 0.20%) 475,033k 475,101k p=0.005 n=6
Parse Time 3.68s (± 0.14%) 3.29s (± 0.23%) 🟩-0.39s (-10.47%) 3.28s 3.30s p=0.004 n=6
Bind Time 1.02s (± 0.40%) 0.97s (± 0.78%) 🟩-0.05s (- 4.91%) 0.96s 0.98s p=0.003 n=6
Check Time 18.21s (± 0.67%) 17.91s (± 0.37%) -0.30s (- 1.66%) 17.81s 17.97s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 22.91s (± 0.54%) 22.17s (± 0.26%) 🟩-0.74s (- 3.22%) 22.08s 22.23s p=0.005 n=6
xstate - node (v16.17.1, x64)
Memory used 546,665k (± 0.01%) 544,187k (± 0.02%) -2,477k (- 0.45%) 544,121k 544,373k p=0.005 n=6
Parse Time 4.77s (± 0.22%) 4.22s (± 0.30%) 🟩-0.55s (-11.47%) 4.20s 4.23s p=0.004 n=6
Bind Time 1.84s (± 0.28%) 1.73s (± 0.44%) 🟩-0.11s (- 6.24%) 1.72s 1.74s p=0.004 n=6
Check Time 3.06s (± 0.65%) 2.97s (± 0.56%) -0.09s (- 2.94%) 2.94s 2.99s p=0.005 n=6
Emit Time 0.09s (± 4.45%) 0.09s (± 0.00%) ~ 0.09s 0.09s p=0.405 n=6
Total Time 9.76s (± 0.29%) 9.01s (± 0.33%) 🟩-0.75s (- 7.68%) 8.97s 9.04s p=0.005 n=6
Angular - node (v14.15.1, x64)
Memory used 352,818k (± 0.01%) 351,944k (± 0.00%) -874k (- 0.25%) 351,918k 351,956k p=0.005 n=6
Parse Time 4.23s (± 0.35%) 3.74s (± 0.53%) 🟩-0.49s (-11.55%) 3.72s 3.77s p=0.005 n=6
Bind Time 1.28s (± 0.80%) 1.25s (± 0.51%) -0.03s (- 2.60%) 1.24s 1.26s p=0.004 n=6
Check Time 9.74s (± 0.71%) 9.66s (± 0.33%) ~ 9.62s 9.70s p=0.065 n=6
Emit Time 8.37s (± 0.54%) 8.20s (± 0.64%) -0.17s (- 2.01%) 8.16s 8.30s p=0.006 n=6
Total Time 23.61s (± 0.42%) 22.85s (± 0.22%) 🟩-0.77s (- 3.24%) 22.77s 22.92s p=0.005 n=6
Compiler-Unions - node (v14.15.1, x64)
Memory used 188,784k (± 0.05%) 188,435k (± 0.01%) -348k (- 0.18%) 188,417k 188,455k p=0.005 n=6
Parse Time 1.83s (± 0.41%) 1.59s (± 0.51%) 🟩-0.25s (-13.38%) 1.58s 1.60s p=0.004 n=6
Bind Time 0.87s (± 0.97%) 0.85s (± 0.61%) -0.02s (- 2.12%) 0.84s 0.85s p=0.004 n=6
Check Time 10.50s (± 0.59%) 10.26s (± 0.37%) -0.24s (- 2.25%) 10.19s 10.29s p=0.005 n=6
Emit Time 3.39s (± 3.89%) 3.08s (± 0.93%) 🟩-0.31s (- 9.14%) 3.04s 3.12s p=0.005 n=6
Total Time 16.59s (± 1.03%) 15.78s (± 0.32%) 🟩-0.81s (- 4.89%) 15.70s 15.85s p=0.005 n=6
Monaco - node (v14.15.1, x64)
Memory used 337,993k (± 0.00%) 337,278k (± 0.01%) -715k (- 0.21%) 337,262k 337,310k p=0.005 n=6
Parse Time 3.20s (± 0.67%) 2.87s (± 0.36%) 🟩-0.33s (-10.16%) 2.86s 2.89s p=0.005 n=6
Bind Time 1.11s (± 0.46%) 1.12s (± 0.49%) ~ 1.11s 1.12s p=0.640 n=6
Check Time 8.11s (± 0.56%) 7.96s (± 0.81%) -0.15s (- 1.85%) 7.86s 8.03s p=0.005 n=6
Emit Time 4.80s (± 0.83%) 4.55s (± 0.98%) 🟩-0.25s (- 5.27%) 4.49s 4.60s p=0.005 n=6
Total Time 17.23s (± 0.34%) 16.50s (± 0.51%) 🟩-0.73s (- 4.22%) 16.33s 16.56s p=0.005 n=6
TFS - node (v14.15.1, x64)
Memory used 294,575k (± 0.01%) 293,924k (± 0.01%) -651k (- 0.22%) 293,884k 293,941k p=0.005 n=6
Parse Time 2.74s (± 0.68%) 2.42s (± 0.48%) 🟩-0.32s (-11.78%) 2.41s 2.44s p=0.005 n=6
Bind Time 1.11s (± 0.95%) 1.08s (± 0.50%) -0.02s (- 1.81%) 1.08s 1.09s p=0.008 n=6
Check Time 7.51s (± 0.47%) 7.45s (± 0.40%) -0.07s (- 0.93%) 7.41s 7.48s p=0.013 n=6
Emit Time 4.63s (± 1.03%) 4.29s (± 1.01%) 🟩-0.34s (- 7.31%) 4.21s 4.33s p=0.005 n=6
Total Time 15.99s (± 0.44%) 15.24s (± 0.43%) 🟩-0.75s (- 4.69%) 15.12s 15.31s p=0.005 n=6
material-ui - node (v14.15.1, x64)
Memory used 471,622k (± 0.00%) 470,749k (± 0.01%) -873k (- 0.19%) 470,722k 470,778k p=0.005 n=6
Parse Time 3.88s (± 0.72%) 3.44s (± 0.61%) 🟩-0.44s (-11.26%) 3.42s 3.48s p=0.005 n=6
Bind Time 1.05s (± 0.00%) 1.01s (± 0.40%) 🟩-0.04s (- 3.97%) 1.00s 1.01s p=0.002 n=6
Check Time 19.14s (± 0.77%) 18.92s (± 0.31%) -0.22s (- 1.16%) 18.82s 18.99s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 24.07s (± 0.57%) 23.37s (± 0.23%) -0.70s (- 2.89%) 23.27s 23.42s p=0.005 n=6
xstate - node (v14.15.1, x64)
Memory used 534,749k (± 0.00%) 533,029k (± 0.02%) -1,721k (- 0.32%) 532,837k 533,091k p=0.005 n=6
Parse Time 5.12s (± 0.82%) 4.36s (± 0.23%) 🟩-0.76s (-14.87%) 4.35s 4.37s p=0.005 n=6
Bind Time 1.71s (± 0.60%) 1.76s (± 0.78%) +0.06s (+ 3.42%) 1.74s 1.78s p=0.004 n=6
Check Time 3.13s (± 0.48%) 3.14s (± 0.70%) ~ 3.11s 3.17s p=0.256 n=6
Emit Time 0.10s (± 4.01%) 0.09s (± 0.00%) 🟩-0.01s (-11.48%) 0.09s 0.09s p=0.002 n=6
Total Time 10.06s (± 0.36%) 9.36s (± 0.34%) 🟩-0.70s (- 6.96%) 9.32s 9.40s p=0.005 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.15.1, x64)
Scenarios
  • Angular - node (v18.10.0, x64)
  • Angular - node (v16.17.1, x64)
  • Angular - node (v14.15.1, x64)
  • Compiler-Unions - node (v18.10.0, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Compiler-Unions - node (v14.15.1, x64)
  • Monaco - node (v18.10.0, x64)
  • Monaco - node (v16.17.1, x64)
  • Monaco - node (v14.15.1, x64)
  • TFS - node (v18.10.0, x64)
  • TFS - node (v16.17.1, x64)
  • TFS - node (v14.15.1, x64)
  • material-ui - node (v18.10.0, x64)
  • material-ui - node (v16.17.1, x64)
  • material-ui - node (v14.15.1, x64)
  • xstate - node (v18.10.0, x64)
  • xstate - node (v16.17.1, x64)
  • xstate - node (v14.15.1, x64)
Benchmark Name Iterations
Current 52656 6
Baseline main 6

TSServer

Comparison Report - main..52656
Metric main 52656 Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,496ms (± 1.28%) 2,378ms (± 1.33%) 🟩-118ms (- 4.73%) 2,338ms 2,422ms p=0.005 n=6
Req 2 - geterr 5,511ms (± 0.57%) 5,413ms (± 0.51%) -98ms (- 1.77%) 5,375ms 5,452ms p=0.005 n=6
Req 3 - references 370ms (± 1.13%) 345ms (± 1.33%) 🟩-25ms (- 6.77%) 340ms 353ms p=0.005 n=6
Req 4 - navto 284ms (± 0.99%) 277ms (± 0.90%) -7ms (- 2.52%) 272ms 279ms p=0.005 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 73ms (± 1.03%) 69ms (± 5.72%) ~ 67ms 77ms p=0.061 n=6
CompilerTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 2,739ms (± 0.79%) 2,494ms (± 0.89%) 🟩-245ms (- 8.93%) 2,459ms 2,522ms p=0.005 n=6
Req 2 - geterr 4,059ms (± 0.64%) 3,990ms (± 0.63%) -69ms (- 1.70%) 3,959ms 4,035ms p=0.008 n=6
Req 3 - references 368ms (± 0.72%) 362ms (± 0.69%) -6ms (- 1.63%) 359ms 366ms p=0.013 n=6
Req 4 - navto 293ms (± 0.88%) 293ms (± 0.91%) ~ 290ms 298ms p=1.000 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 66ms (± 0.79%) 66ms (± 9.84%) ~ 62ms 77ms p=0.366 n=6
xstateTSServer - node (v18.10.0, x64)
Req 1 - updateOpen 3,490ms (± 0.82%) 3,181ms (± 0.70%) 🟩-309ms (- 8.85%) 3,145ms 3,204ms p=0.005 n=6
Req 2 - geterr 1,648ms (± 0.66%) 1,682ms (± 1.35%) +35ms (+ 2.09%) 1,657ms 1,711ms p=0.005 n=6
Req 3 - references 104ms (± 1.05%) 103ms (± 2.07%) ~ 101ms 106ms p=0.566 n=6
Req 4 - navto 355ms (± 0.53%) 367ms (± 0.93%) +13ms (+ 3.57%) 363ms 373ms p=0.005 n=6
Req 5 - completionInfo count 3,175 (± 0.00%) 3,175 (± 0.00%) ~ 3,175 3,175 p=1.000 n=6
Req 5 - completionInfo 450ms (± 0.90%) 445ms (± 1.17%) ~ 439ms 452ms p=0.092 n=6
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,770ms (± 0.58%) 2,529ms (± 0.91%) 🟩-242ms (- 8.72%) 2,496ms 2,554ms p=0.005 n=6
Req 2 - geterr 5,930ms (± 0.53%) 5,774ms (± 0.36%) -156ms (- 2.63%) 5,743ms 5,800ms p=0.005 n=6
Req 3 - references 377ms (± 0.76%) 360ms (± 0.49%) 🟩-18ms (- 4.64%) 357ms 362ms p=0.005 n=6
Req 4 - navto 277ms (± 0.91%) 265ms (± 0.67%) 🟩-12ms (- 4.33%) 263ms 268ms p=0.005 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 79ms (± 2.70%) 88ms (± 2.42%) +9ms (+11.37%) 84ms 90ms p=0.005 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,924ms (± 0.37%) 2,654ms (± 1.00%) 🟩-270ms (- 9.23%) 2,615ms 2,692ms p=0.005 n=6
Req 2 - geterr 4,412ms (± 0.53%) 4,332ms (± 0.38%) -81ms (- 1.82%) 4,308ms 4,352ms p=0.005 n=6
Req 3 - references 382ms (± 1.07%) 376ms (± 0.48%) -6ms (- 1.66%) 373ms 378ms p=0.036 n=6
Req 4 - navto 287ms (± 0.56%) 284ms (± 0.36%) -4ms (- 1.22%) 282ms 285ms p=0.004 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 71ms (± 0.57%) 67ms (± 1.26%) 🟩-5ms (- 6.56%) 65ms 67ms p=0.003 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 3,626ms (± 0.35%) 3,341ms (± 0.52%) 🟩-285ms (- 7.86%) 3,319ms 3,362ms p=0.005 n=6
Req 2 - geterr 1,825ms (± 1.51%) 2,014ms (± 2.38%) +189ms (+10.37%) 1,923ms 2,049ms p=0.005 n=6
Req 3 - references 113ms (± 1.42%) 110ms (± 2.07%) -3ms (- 2.51%) 106ms 112ms p=0.034 n=6
Req 4 - navto 344ms (± 1.02%) 355ms (± 0.95%) +11ms (+ 3.15%) 349ms 358ms p=0.006 n=6
Req 5 - completionInfo count 3,175 (± 0.00%) 3,175 (± 0.00%) ~ 3,175 3,175 p=1.000 n=6
Req 5 - completionInfo 457ms (± 0.99%) 450ms (± 1.43%) ~ 443ms 462ms p=0.064 n=6
Compiler-UnionsTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 2,884ms (± 0.28%) 2,589ms (± 0.51%) 🟩-295ms (-10.22%) 2,572ms 2,604ms p=0.005 n=6
Req 2 - geterr 6,277ms (± 0.73%) 6,207ms (± 0.59%) -70ms (- 1.11%) 6,156ms 6,250ms p=0.020 n=6
Req 3 - references 391ms (± 0.53%) 368ms (± 0.48%) 🟩-23ms (- 5.85%) 365ms 370ms p=0.005 n=6
Req 4 - navto 280ms (± 1.31%) 279ms (± 0.67%) ~ 278ms 283ms p=0.933 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 99ms (± 1.69%) 98ms (± 2.29%) ~ 94ms 100ms p=0.676 n=6
CompilerTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 3,038ms (± 0.96%) 2,772ms (± 0.97%) 🟩-266ms (- 8.75%) 2,735ms 2,811ms p=0.005 n=6
Req 2 - geterr 4,695ms (± 0.40%) 4,642ms (± 1.86%) ~ 4,468ms 4,700ms p=0.093 n=6
Req 3 - references 404ms (± 0.43%) 388ms (± 0.53%) 🟩-16ms (- 3.96%) 386ms 391ms p=0.005 n=6
Req 4 - navto 293ms (± 1.30%) 287ms (± 1.15%) -6ms (- 1.94%) 284ms 293ms p=0.035 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 82ms (± 3.84%) 82ms (± 7.67%) ~ 79ms 95ms p=0.230 n=6
xstateTSServer - node (v14.15.1, x64)
Req 1 - updateOpen 3,955ms (± 0.60%) 3,628ms (± 0.52%) 🟩-327ms (- 8.27%) 3,600ms 3,655ms p=0.005 n=6
Req 2 - geterr 1,776ms (± 1.68%) 1,804ms (± 0.78%) ~ 1,785ms 1,825ms p=0.066 n=6
Req 3 - references 133ms (± 1.99%) 124ms (± 1.56%) 🟩-9ms (- 6.88%) 121ms 126ms p=0.005 n=6
Req 4 - navto 375ms (± 0.95%) 383ms (± 0.47%) +8ms (+ 2.13%) 381ms 386ms p=0.005 n=6
Req 5 - completionInfo count 3,175 (± 0.00%) 3,175 (± 0.00%) ~ 3,175 3,175 p=1.000 n=6
Req 5 - completionInfo 470ms (± 2.40%) 461ms (± 0.60%) ~ 457ms 465ms p=0.297 n=6
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v18.10.0, x64)
  • node (v16.17.1, x64)
  • node (v14.15.1, x64)
Scenarios
  • Compiler-UnionsTSServer - node (v18.10.0, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v14.15.1, x64)
  • CompilerTSServer - node (v18.10.0, x64)
  • CompilerTSServer - node (v16.17.1, x64)
  • CompilerTSServer - node (v14.15.1, x64)
  • xstateTSServer - node (v18.10.0, x64)
  • xstateTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v14.15.1, x64)
Benchmark Name Iterations
Current 52656 6
Baseline main 6

Startup

Comparison Report - main..52656
Metric main 52656 Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 142.53ms (± 0.21%) 140.66ms (± 0.18%) -1.87ms (- 1.31%) 140.03ms 143.81ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 227.47ms (± 0.18%) 223.95ms (± 0.18%) -3.52ms (- 1.55%) 222.71ms 227.24ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 228.98ms (± 0.13%) 228.47ms (± 0.41%) -0.51ms (- 0.22%) 225.12ms 239.95ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 209.74ms (± 0.15%) 209.00ms (± 0.37%) -0.74ms (- 0.35%) 206.25ms 214.83ms p=0.000 n=600
System
Machine Namets-ci-ubuntu
Platformlinux 5.4.0-135-generic
Architecturex64
Available Memory16 GB
Available Memory15 GB
CPUs4 × Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current 52656 6
Baseline main 6

Developer Information:

Download Benchmark

@Andarist
Copy link
Contributor

Andarist commented Feb 7, 2023

Not bad results 😉

@jakebailey
Copy link
Member Author

Yeah, though this was already known when we merged modules and gave up var downleveling. Really unfortunate that this helps so much; I'm trying to figure out if there's some subset of variables that can be changed or if it's just a million papercuts.

@Andarist
Copy link
Contributor

Andarist commented Feb 7, 2023 •
edited

What’s the concern here though? Just the build time?

@jakebailey
Copy link
Member Author

jakebailey commented Feb 7, 2023 •
edited

Well, I wouldn't merge this as-is because (in no particular order):

  • It introduces another third-party dep into the critical build chain, which right now is solely tsc and esbuild, and we have the ability to drop the latter if needed.
    • Though, arguably, the latter wouldn't be great for performance either, so what's the point in avoiding it?
  • It slows down the build, and I don't really want to re-introduce the idea of a "release" build of TypeScript.
    • But, we can just keep let/const for tests and catch violations not already caught by TS that way.
  • Loops get rewritten, which makes the files annoying to use while debugging; preferrably there would be a performance-focused variant of the transform which only did the "easy" ones.
    • Again, could be non-test only.
  • babel deoptimizes the files because they're so big, which effectively makes them as hard to patch as minified versions of TypeScript, and we have an unreasonable number of people who do that. I half-mitigated this by rerunning the output through esbuild, though that was to try and avoid the perf diff showing a startup speed boots from the simple fact that the files are smaller.
    • But, it could be implemented in esbuild directly.
  • Long term, I think it'd be bad to be ESM but have export var (which I believe is slower for consumers because they are live bindings and not const).

@jakebailey
Copy link
Member Author

It slows down the build

On the whole though, the "big" build tasks don't actually get longer because esbuild/babel are running in parallel with dts emit / bundling, but the actual bundle tasks go from half a second to 8 seconds, which isn't great for the debugging loop.

This was referenced Feb 7, 2023
Closed
Closed
@fatcerberus
Copy link

fatcerberus commented Feb 7, 2023 •
edited

I assume the performance hit for let/const vs. var is down to TDZ tracking. Which is unfortunate because var has enough footguns that I’d never use it directly in a codebase and I’m not running babel over my code as a build step just to change all the lets to vars.

@jakebailey
Copy link
Member Author

Yep, it's TDZ. I'm trying to figure out if there's a limited subset that can be changed to make parsing faster but it's not really turning out well.

@fatcerberus
Copy link

It’s weird because most potential TDZ violations can be detected syntactically so engines should have figured out how to optimize it away by now.

@jakebailey
Copy link
Member Author

I would certainly think. But at this point I'm having a hard time justifying not doing some form of this change.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Feb 7, 2023
@jakebailey
Copy link
Member Author

jakebailey commented Feb 7, 2023 •
edited

If people are stumbling on this, last time I tested this, it was 5-9%, and now it's more like 7-15%; the relative proportion has changed because we've increased the performance of the TS compiler since modules has merged meaning the fixed perf bump here is now a bigger overall proportion.

@jakebailey
Copy link
Member Author

I hacked up TypeScript and ts-perf to run this PR against bun v0.5.5. It doesn't seem like the engine Bun uses (Apple's JavaScriptCore) is affected by the TDZ problem; I'd chalk up any perf differences below to noise.

Note that these benchmark results are not comparable to the ones earlier in this thread; I performed these tests on an entirely different perf-tuned machine in my office, not the corp-owned machine that we normally use. But, I probably will try and upstream my hacks to ts-perf as I've been wanting to test some more interesting hosts.

Details

Loading benchmark and comparing to baseline.
Comparison Report - main..HEAD

Metric main HEAD Delta Best Worst p-value
Angular - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.57s (± 0.53%) 2.53s (± 2.39%) ~ 2.49s 2.65s p=0.064 n=6
Bind Time 1.05s (± 0.85%) 1.05s (± 1.55%) ~ 1.03s 1.08s p=0.801 n=6
Check Time 9.85s (± 0.58%) 9.81s (± 0.49%) ~ 9.76s 9.89s p=0.261 n=6
Emit Time 8.69s (± 0.51%) 8.58s (± 0.49%) -0.11s (- 1.27%) 8.51s 8.63s p=0.008 n=6
Total Time 22.15s (± 0.31%) 21.98s (± 0.54%) -0.18s (- 0.80%) 21.82s 22.17s p=0.013 n=6
Compiler - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.47s (± 0.93%) 1.51s (± 1.37%) +0.03s (+ 2.26%) 1.49s 1.54s p=0.009 n=6
Bind Time 1.03s (± 1.13%) 1.04s (± 1.16%) ~ 1.02s 1.05s p=0.131 n=6
Check Time 8.49s (± 0.72%) 8.45s (± 0.40%) ~ 8.41s 8.49s p=0.332 n=6
Emit Time 3.68s (± 0.96%) 3.61s (± 0.65%) -0.07s (- 1.81%) 3.59s 3.64s p=0.005 n=6
Total Time 14.67s (± 0.53%) 14.62s (± 0.30%) ~ 14.57s 14.68s p=0.297 n=6
Compiler-Unions - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.43s (± 2.27%) 1.41s (± 2.00%) ~ 1.36s 1.43s p=0.346 n=6
Bind Time 0.98s (± 0.77%) 0.96s (± 2.68%) ~ 0.94s 1.01s p=0.191 n=6
Check Time 11.62s (± 0.89%) 11.69s (± 1.19%) ~ 11.50s 11.85s p=0.377 n=6
Emit Time 3.44s (± 1.07%) 3.39s (± 1.48%) ~ 3.33s 3.47s p=0.106 n=6
Total Time 17.46s (± 0.46%) 17.45s (± 0.72%) ~ 17.29s 17.63s p=0.936 n=6
Monaco - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.41s (± 2.65%) 2.35s (± 3.28%) -0.06s (- 2.35%) 2.31s 2.51s p=0.044 n=6
Bind Time 1.35s (± 1.89%) 1.36s (± 1.90%) ~ 1.32s 1.39s p=0.872 n=6
Check Time 8.69s (± 0.35%) 8.67s (± 0.31%) ~ 8.65s 8.72s p=0.406 n=6
Emit Time 4.75s (± 1.27%) 4.72s (± 1.57%) ~ 4.64s 4.84s p=0.376 n=6
Total Time 17.20s (± 0.65%) 17.10s (± 1.05%) ~ 16.96s 17.45s p=0.090 n=6
TFS - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.08s (± 1.41%) 2.05s (± 1.29%) ~ 2.03s 2.10s p=0.106 n=6
Bind Time 1.16s (± 2.59%) 1.17s (± 1.13%) ~ 1.15s 1.19s p=0.220 n=6
Check Time 8.04s (± 0.36%) 8.08s (± 0.52%) ~ 8.04s 8.15s p=0.087 n=6
Emit Time 4.35s (± 0.55%) 4.27s (± 0.84%) -0.08s (- 1.80%) 4.21s 4.32s p=0.008 n=6
Total Time 15.62s (± 0.18%) 15.58s (± 0.34%) ~ 15.51s 15.65s p=0.173 n=6
material-ui - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.75s (± 1.58%) 2.75s (± 2.03%) ~ 2.71s 2.86s p=0.684 n=6
Bind Time 0.92s (± 3.60%) 0.94s (± 0.55%) ~ 0.94s 0.95s p=0.210 n=6
Check Time 21.98s (± 1.38%) 21.26s (± 0.65%) 🟩-0.72s (- 3.27%) 21.00s 21.40s p=0.008 n=6
Emit Time 0.00s (±244.70%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.405 n=6
Total Time 25.66s (± 1.32%) 24.96s (± 0.55%) -0.70s (- 2.72%) 24.70s 25.09s p=0.020 n=6
xstate - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 4.15s (± 0.72%) 4.03s (± 0.30%) 🟩-0.13s (- 3.01%) 4.01s 4.04s p=0.005 n=6
Bind Time 1.58s (± 0.89%) 1.61s (± 0.94%) +0.03s (+ 1.69%) 1.59s 1.63s p=0.018 n=6
Check Time 4.03s (± 0.44%) 4.08s (± 0.51%) +0.05s (+ 1.24%) 4.06s 4.11s p=0.010 n=6
Emit Time 0.23s (± 2.28%) 0.22s (± 0.00%) -0.01s (- 2.94%) 0.22s 0.22s p=0.025 n=6
Total Time 9.99s (± 0.32%) 9.94s (± 0.24%) -0.05s (- 0.52%) 9.90s 9.96s p=0.023 n=6
System
Machine Nameelcheapo
Platformlinux 6.1.9-arch1-2
Architecturex64
Available Memory8 GB
Available Memory5 GB
CPUs6 × Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
Hosts
  • bun (v0.5.5, x64)
Scenarios
  • Angular - bun (v0.5.5, x64)
  • Compiler - bun (v0.5.5, x64)
  • Compiler-Unions - bun (v0.5.5, x64)
  • Monaco - bun (v0.5.5, x64)
  • TFS - bun (v0.5.5, x64)
  • material-ui - bun (v0.5.5, x64)
  • xstate - bun (v0.5.5, x64)
Benchmark Name Iterations
Current HEAD 6
Baseline main 6

@jakebailey
Copy link
Member Author

But, i'm going to rerun this just to be sure I didn't mess up somehow.

@jakebailey
Copy link
Member Author

jakebailey commented Feb 8, 2023 •
edited

Nah, seems correct. A second go:

Details

Loading benchmark and comparing to baseline.
Comparison Report - main..HEAD

Metric main HEAD Delta Best Worst p-value
Angular - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.57s (± 0.98%) 2.54s (± 2.64%) ~ 2.50s 2.67s p=0.062 n=6
Bind Time 1.06s (± 1.10%) 1.07s (± 3.01%) ~ 1.04s 1.13s p=0.806 n=6
Check Time 9.80s (± 0.54%) 9.84s (± 0.68%) ~ 9.77s 9.96s p=0.573 n=6
Emit Time 8.71s (± 1.07%) 8.62s (± 0.93%) ~ 8.55s 8.76s p=0.128 n=6
Total Time 22.14s (± 0.37%) 22.07s (± 0.71%) ~ 21.86s 22.30s p=0.471 n=6
Compiler - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.47s (± 1.27%) 1.51s (± 1.62%) +0.04s (+ 2.95%) 1.49s 1.55s p=0.011 n=6
Bind Time 1.03s (± 1.50%) 1.03s (± 1.84%) ~ 1.01s 1.06s p=0.799 n=6
Check Time 8.44s (± 0.59%) 8.47s (± 0.66%) ~ 8.41s 8.56s p=0.261 n=6
Emit Time 3.67s (± 0.80%) 3.63s (± 0.73%) ~ 3.59s 3.66s p=0.077 n=6
Total Time 14.60s (± 0.53%) 14.65s (± 0.44%) ~ 14.56s 14.73s p=0.298 n=6
Compiler-Unions - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 1.42s (± 1.06%) 1.41s (± 3.28%) ~ 1.37s 1.49s p=0.368 n=6
Bind Time 0.97s (± 0.53%) 0.96s (± 1.70%) ~ 0.94s 0.98s p=0.542 n=6
Check Time 11.66s (± 0.40%) 11.63s (± 0.91%) ~ 11.47s 11.77s p=0.810 n=6
Emit Time 3.43s (± 1.19%) 3.35s (± 1.50%) -0.08s (- 2.38%) 3.29s 3.40s p=0.015 n=6
Total Time 17.47s (± 0.38%) 17.36s (± 0.40%) -0.12s (- 0.67%) 17.25s 17.43s p=0.019 n=6
Monaco - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.39s (± 0.57%) 2.35s (± 3.22%) ~ 2.30s 2.50s p=0.060 n=6
Bind Time 1.35s (± 5.27%) 1.33s (± 7.58%) ~ 1.13s 1.40s p=0.870 n=6
Check Time 8.70s (± 1.09%) 8.70s (± 0.83%) ~ 8.64s 8.83s p=0.809 n=6
Emit Time 4.75s (± 0.77%) 4.71s (± 0.46%) ~ 4.68s 4.74s p=0.072 n=6
Total Time 17.18s (± 0.42%) 17.08s (± 0.53%) ~ 17.01s 17.22s p=0.065 n=6
TFS - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.07s (± 0.66%) 2.05s (± 1.91%) ~ 2.03s 2.13s p=0.075 n=6
Bind Time 1.17s (± 2.28%) 1.17s (± 0.44%) ~ 1.17s 1.18s p=0.667 n=6
Check Time 8.02s (± 0.38%) 8.08s (± 0.26%) +0.06s (+ 0.79%) 8.05s 8.10s p=0.006 n=6
Emit Time 4.33s (± 1.72%) 4.26s (± 1.05%) ~ 4.20s 4.32s p=0.054 n=6
Total Time 15.58s (± 0.57%) 15.56s (± 0.37%) ~ 15.49s 15.65s p=0.873 n=6
material-ui - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 2.75s (± 0.61%) 2.76s (± 2.06%) ~ 2.72s 2.87s p=0.806 n=6
Bind Time 0.93s (± 4.07%) 0.95s (± 0.79%) ~ 0.94s 0.96s p=0.081 n=6
Check Time 22.12s (± 0.61%) 21.37s (± 0.38%) 🟩-0.75s (- 3.39%) 21.27s 21.51s p=0.005 n=6
Emit Time 0.00s (±244.70%) 0.00s (±244.70%) ~ 0.00s 0.01s p=1.000 n=6
Total Time 25.80s (± 0.63%) 25.09s (± 0.40%) -0.71s (- 2.75%) 24.97s 25.23s p=0.005 n=6
xstate - bun (v0.5.5, x64)
Memory used 0k (± 0.00%) 0k (± 0.00%) ~ 0k 0k p=1.000 n=6
Parse Time 4.15s (± 0.20%) 4.02s (± 0.72%) 🟩-0.13s (- 3.10%) 3.99s 4.07s p=0.005 n=6
Bind Time 1.57s (± 1.32%) 1.63s (± 1.08%) +0.05s (+ 3.17%) 1.60s 1.64s p=0.006 n=6
Check Time 4.03s (± 0.30%) 4.07s (± 0.39%) +0.04s (+ 0.95%) 4.06s 4.10s p=0.005 n=6
Emit Time 0.23s (± 3.88%) 0.22s (± 1.84%) ~ 0.22s 0.23s p=0.086 n=6
Total Time 9.99s (± 0.22%) 9.94s (± 0.45%) -0.05s (- 0.50%) 9.89s 10.01s p=0.044 n=6
System
Machine Nameelcheapo
Platformlinux 6.1.9-arch1-2
Architecturex64
Available Memory8 GB
Available Memory5 GB
CPUs6 × Intel(R) Core(TM) i5-8400 CPU @ 2.80GHz
Hosts
  • bun (v0.5.5, x64)
Scenarios
  • Angular - bun (v0.5.5, x64)
  • Compiler - bun (v0.5.5, x64)
  • Compiler-Unions - bun (v0.5.5, x64)
  • Monaco - bun (v0.5.5, x64)
  • TFS - bun (v0.5.5, x64)
  • material-ui - bun (v0.5.5, x64)
  • xstate - bun (v0.5.5, x64)
Benchmark Name Iterations
Current HEAD 6
Baseline main 6

EDIT: Updated this table to say "bun"; I did that in the last one but forgot in this one. ts-perf doesn't currently support bun so this is a hack.

@Jarred-Sumner
Copy link

Jarred-Sumner commented Feb 8, 2023 •
edited

Note that Bun runs the transpiler on every file loaded either via import or require (not eval, not new Function, etc). It doesn’t convert let/const variables to var. I wouldn’t expect the results to be different l. it might not work, but you could try BUN_DISABLE_TRANSPILER=1 & it would rule out the possibility of Bun’s transpiler changing the output before being evaluated

@jakebailey
Copy link
Member Author

I'm not sure what you mean; the comparison is main versus this PR, i.e. let/const versus var. So if they don't do this transformation (as you say), then the comparison shows that their engine doesn't have this TDZ performance hit, at least not to any great extent.

@jakebailey
Copy link
Member Author

Unless you mean that I should try disabling the transpiler to make sure 100% they don't already do this transformation.

@Jarred-Sumner
Copy link

Jarred-Sumner commented Feb 8, 2023 •
edited

The environment variable which prevents the transpiler from running is here:

https://github.com/oven-sh/bun/blob/main/src/bundler.zig#L539

I think the results were already correct, just that since it’s two transpilers running it’s harder to be 100% sure. I sent the above message without re-reading the words I wrote very carefully. Bun doesn’t convert let or const to var, but does a bunch of other stuff. I do know that ESM imports bindings have virtually no overhead in JSC (there was a thread in Node’s issues about ESM imports being slower than var at some point)

@marvinhagemeister
Copy link

Looping in @evanw because that may be an interesting optimization to do right in esbuild itself.

@jakebailey
Copy link
Member Author

Looping in @evanw because that may be an interesting optimization to do right in esbuild itself.

See: evanw/esbuild#2889 (comment)

@Jack-Works
Copy link
Contributor

maybe replace const and let with var or var with simple string replacement? that can avoid the cost of introducing babel.

@jakebailey
Copy link
Member Author

That's been tried before and is too simple to work; there are places where renames are required, along with the loops/switch-cases which have to be rewritten to closures.

To be clear, this isn't a PR I expect to merge at all, I just brought it up to date so I could test the new perf infra and check out the current state of things.

@jakebailey
Copy link
Member Author

@typescript-bot new perf test this faster

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 14, 2023 •
edited

Heya @jakebailey, I've started to run the tsc-only perf test suite on this PR at 5628ee7. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 300,342k (± 0.01%) 299,648k (± 0.00%) -694k (- 0.23%) 299,637k 299,655k p=0.005 n=6
Parse Time 3.02s (± 0.17%) 3.03s (± 0.17%) ~ 3.02s 3.03s p=0.311 n=6
Bind Time 0.93s (± 0.00%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=1.000 n=6
Check Time 9.45s (± 0.24%) 9.39s (± 0.40%) -0.06s (- 0.67%) 9.36s 9.46s p=0.029 n=6
Emit Time 7.64s (± 0.17%) 7.50s (± 0.43%) -0.14s (- 1.85%) 7.47s 7.55s p=0.005 n=6
Total Time 21.05s (± 0.13%) 20.84s (± 0.27%) -0.21s (- 0.97%) 20.78s 20.94s p=0.005 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,834k (± 0.02%) 193,315k (± 0.03%) -519k (- 0.27%) 193,261k 193,389k p=0.005 n=6
Parse Time 1.57s (± 1.06%) 1.57s (± 0.48%) ~ 1.56s 1.58s p=1.000 n=6
Bind Time 0.79s (± 0.00%) 0.78s (± 0.00%) -0.01s (- 1.27%) 0.78s 0.78s p=0.001 n=6
Check Time 9.89s (± 0.30%) 9.80s (± 0.39%) -0.08s (- 0.84%) 9.75s 9.84s p=0.005 n=6
Emit Time 2.74s (± 0.30%) 2.74s (± 0.19%) ~ 2.73s 2.74s p=0.073 n=6
Total Time 14.99s (± 0.19%) 14.89s (± 0.29%) -0.10s (- 0.66%) 14.84s 14.94s p=0.005 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,045k (± 0.00%) 346,519k (± 0.00%) -527k (- 0.15%) 346,506k 346,536k p=0.005 n=6
Parse Time 2.69s (± 0.28%) 2.68s (± 0.24%) -0.01s (- 0.43%) 2.67s 2.69s p=0.027 n=6
Bind Time 0.99s (± 0.41%) 1.00s (± 0.00%) +0.01s (+ 1.18%) 1.00s 1.00s p=0.002 n=6
Check Time 7.91s (± 0.42%) 7.83s (± 0.41%) -0.08s (- 1.03%) 7.78s 7.88s p=0.013 n=6
Emit Time 4.25s (± 0.18%) 4.23s (± 0.28%) -0.02s (- 0.47%) 4.22s 4.25s p=0.017 n=6
Total Time 15.84s (± 0.19%) 15.74s (± 0.30%) -0.11s (- 0.66%) 15.68s 15.82s p=0.013 n=6
TFS - node (v16.17.1, x64)
Memory used 301,124k (± 0.00%) 300,615k (± 0.00%) -509k (- 0.17%) 300,599k 300,623k p=0.005 n=6
Parse Time 2.19s (± 0.94%) 2.16s (± 0.38%) -0.03s (- 1.37%) 2.15s 2.17s p=0.011 n=6
Bind Time 1.11s (± 0.46%) 1.08s (± 1.48%) -0.03s (- 2.84%) 1.07s 1.11s p=0.009 n=6
Check Time 7.19s (± 0.39%) 7.18s (± 0.27%) ~ 7.17s 7.22s p=0.624 n=6
Emit Time 3.97s (± 0.30%) 3.92s (± 0.32%) -0.05s (- 1.18%) 3.90s 3.93s p=0.005 n=6
Total Time 14.46s (± 0.16%) 14.34s (± 0.23%) -0.12s (- 0.81%) 14.31s 14.40s p=0.005 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,449k (± 0.01%) 479,080k (± 0.00%) -369k (- 0.08%) 479,068k 479,097k p=0.005 n=6
Parse Time 3.14s (± 0.16%) 3.17s (± 0.37%) +0.02s (+ 0.80%) 3.16s 3.19s p=0.004 n=6
Bind Time 0.91s (± 0.00%) 0.91s (± 0.45%) ~ 0.91s 0.92s p=0.405 n=6
Check Time 17.87s (± 0.32%) 17.89s (± 0.31%) ~ 17.81s 17.94s p=0.520 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.93s (± 0.25%) 21.97s (± 0.29%) ~ 21.88s 22.04s p=0.335 n=6
xstate - node (v16.17.1, x64)
Memory used 542,778k (± 0.01%) 541,841k (± 0.02%) -937k (- 0.17%) 541,739k 542,011k p=0.005 n=6
Parse Time 3.70s (± 0.20%) 3.66s (± 0.14%) -0.04s (- 1.04%) 3.66s 3.67s p=0.004 n=6
Bind Time 1.38s (± 4.85%) 1.54s (± 3.18%) +0.16s (+11.33%) 1.44s 1.56s p=0.009 n=6
Check Time 3.28s (± 2.74%) 3.16s (± 1.14%) ~ 3.13s 3.21s p=0.053 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 0.00%) ~ 0.08s 0.08s p=1.000 n=6
Total Time 8.45s (± 0.48%) 8.46s (± 0.38%) ~ 8.42s 8.50s p=0.411 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot new perf test this full

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 14, 2023 •
edited

Heya @jakebailey, I've started to run the full perf test suite on this PR at 5628ee7. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v16.17.1, x64)
Memory used 300,338k (± 0.01%) 299,644k (± 0.00%) -694k (- 0.23%) 299,636k 299,653k p=0.005 n=6
Parse Time 3.02s (± 0.40%) 3.02s (± 0.25%) ~ 3.01s 3.03s p=0.242 n=6
Bind Time 0.93s (± 0.00%) 0.93s (± 0.00%) ~ 0.93s 0.93s p=1.000 n=6
Check Time 9.46s (± 0.32%) 9.36s (± 0.26%) -0.10s (- 1.06%) 9.33s 9.39s p=0.005 n=6
Emit Time 7.62s (± 0.26%) 7.48s (± 0.18%) -0.13s (- 1.77%) 7.46s 7.49s p=0.004 n=6
Total Time 21.04s (± 0.26%) 20.79s (± 0.10%) -0.25s (- 1.17%) 20.76s 20.82s p=0.005 n=6
Compiler-Unions - node (v16.17.1, x64)
Memory used 193,813k (± 0.01%) 193,302k (± 0.03%) -512k (- 0.26%) 193,244k 193,414k p=0.005 n=6
Parse Time 1.58s (± 0.33%) 1.57s (± 0.26%) -0.01s (- 0.95%) 1.56s 1.57s p=0.003 n=6
Bind Time 0.79s (± 0.00%) 0.78s (± 0.00%) -0.01s (- 1.27%) 0.78s 0.78s p=0.001 n=6
Check Time 9.87s (± 0.19%) 9.80s (± 0.69%) ~ 9.71s 9.91s p=0.066 n=6
Emit Time 2.75s (± 0.30%) 2.74s (± 0.30%) ~ 2.73s 2.75s p=0.077 n=6
Total Time 14.99s (± 0.16%) 14.89s (± 0.44%) -0.10s (- 0.68%) 14.81s 15.00s p=0.037 n=6
Monaco - node (v16.17.1, x64)
Memory used 347,056k (± 0.00%) 346,513k (± 0.00%) -543k (- 0.16%) 346,504k 346,520k p=0.005 n=6
Parse Time 2.69s (± 0.20%) 2.68s (± 0.20%) -0.02s (- 0.74%) 2.67s 2.68s p=0.004 n=6
Bind Time 0.99s (± 0.00%) 1.00s (± 0.41%) +0.01s (+ 0.84%) 0.99s 1.00s p=0.007 n=6
Check Time 7.89s (± 0.17%) 7.81s (± 0.26%) -0.08s (- 1.01%) 7.79s 7.85s p=0.005 n=6
Emit Time 4.27s (± 0.24%) 4.23s (± 0.12%) -0.03s (- 0.78%) 4.23s 4.24s p=0.004 n=6
Total Time 15.84s (± 0.10%) 15.72s (± 0.13%) -0.13s (- 0.79%) 15.69s 15.75s p=0.004 n=6
TFS - node (v16.17.1, x64)
Memory used 301,122k (± 0.00%) 300,628k (± 0.01%) -494k (- 0.16%) 300,614k 300,658k p=0.005 n=6
Parse Time 2.18s (± 0.84%) 2.15s (± 0.24%) -0.04s (- 1.60%) 2.14s 2.15s p=0.004 n=6
Bind Time 1.09s (± 6.10%) 1.10s (± 1.06%) ~ 1.08s 1.11s p=0.199 n=6
Check Time 7.22s (± 0.12%) 7.18s (± 0.12%) -0.04s (- 0.55%) 7.17s 7.19s p=0.005 n=6
Emit Time 3.98s (± 0.29%) 3.94s (± 0.27%) -0.04s (- 1.09%) 3.92s 3.95s p=0.005 n=6
Total Time 14.47s (± 0.58%) 14.36s (± 0.10%) ~ 14.35s 14.39s p=0.064 n=6
material-ui - node (v16.17.1, x64)
Memory used 479,405k (± 0.01%) 479,087k (± 0.00%) -318k (- 0.07%) 479,079k 479,098k p=0.005 n=6
Parse Time 3.15s (± 0.00%) 3.17s (± 0.24%) +0.02s (+ 0.58%) 3.16s 3.18s p=0.003 n=6
Bind Time 0.91s (± 0.00%) 0.91s (± 0.00%) ~ 0.91s 0.91s p=1.000 n=6
Check Time 17.93s (± 0.21%) 17.93s (± 0.38%) ~ 17.85s 18.01s p=1.000 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 21.99s (± 0.18%) 22.01s (± 0.31%) ~ 21.93s 22.08s p=0.630 n=6
xstate - node (v16.17.1, x64)
Memory used 542,814k (± 0.01%) 541,825k (± 0.01%) -989k (- 0.18%) 541,730k 541,884k p=0.005 n=6
Parse Time 3.69s (± 0.20%) 3.67s (± 0.15%) -0.03s (- 0.72%) 3.66s 3.67s p=0.004 n=6
Bind Time 1.44s (± 3.46%) 1.56s (± 0.41%) +0.12s (+ 8.21%) 1.55s 1.57s p=0.004 n=6
Check Time 3.19s (± 1.56%) 3.17s (± 0.85%) ~ 3.13s 3.21s p=0.629 n=6
Emit Time 0.08s (± 0.00%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.405 n=6
Total Time 8.41s (± 0.22%) 8.48s (± 0.28%) +0.08s (+ 0.93%) 8.45s 8.52s p=0.005 n=6
Angular - node (v18.15.0, x64)
Memory used 301,119k (± 0.00%) 300,472k (± 0.00%) -647k (- 0.21%) 300,447k 300,485k p=0.005 n=6
Parse Time 2.85s (± 0.36%) 2.84s (± 0.61%) ~ 2.82s 2.86s p=0.182 n=6
Bind Time 0.85s (± 0.64%) 0.86s (± 0.00%) ~ 0.86s 0.86s p=0.071 n=6
Check Time 8.23s (± 0.20%) 8.08s (± 0.29%) -0.14s (- 1.74%) 8.05s 8.12s p=0.005 n=6
Emit Time 7.13s (± 0.33%) 7.01s (± 0.26%) -0.12s (- 1.71%) 6.99s 7.03s p=0.005 n=6
Total Time 19.07s (± 0.18%) 18.79s (± 0.15%) -0.28s (- 1.46%) 18.76s 18.84s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 194,509k (± 1.40%) 194,088k (± 1.43%) ~ 191,705k 197,511k p=0.298 n=6
Parse Time 1.49s (± 1.11%) 1.48s (± 1.02%) ~ 1.46s 1.49s p=0.314 n=6
Bind Time 0.74s (± 0.00%) 0.72s (± 0.00%) -0.02s (- 2.70%) 0.72s 0.72s p=0.001 n=6
Check Time 9.15s (± 0.56%) 8.98s (± 0.53%) -0.18s (- 1.93%) 8.93s 9.05s p=0.005 n=6
Emit Time 2.53s (± 0.78%) 2.53s (± 1.57%) ~ 2.49s 2.60s p=0.373 n=6
Total Time 13.91s (± 0.37%) 13.70s (± 0.32%) -0.21s (- 1.49%) 13.65s 13.78s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,874k (± 0.01%) 347,360k (± 0.01%) -514k (- 0.15%) 347,319k 347,386k p=0.005 n=6
Parse Time 2.51s (± 0.50%) 2.51s (± 0.54%) ~ 2.50s 2.54s p=1.000 n=6
Bind Time 0.93s (± 0.55%) 0.93s (± 0.68%) ~ 0.92s 0.94s p=0.386 n=6
Check Time 6.85s (± 0.55%) 6.77s (± 0.35%) -0.08s (- 1.10%) 6.75s 6.81s p=0.012 n=6
Emit Time 4.08s (± 0.61%) 4.04s (± 0.34%) -0.04s (- 1.06%) 4.02s 4.05s p=0.016 n=6
Total Time 14.37s (± 0.53%) 14.24s (± 0.29%) -0.13s (- 0.87%) 14.20s 14.30s p=0.024 n=6
TFS - node (v18.15.0, x64)
Memory used 301,959k (± 0.01%) 301,452k (± 0.01%) -506k (- 0.17%) 301,411k 301,533k p=0.005 n=6
Parse Time 2.04s (± 0.25%) 2.04s (± 0.37%) ~ 2.03s 2.05s p=0.241 n=6
Bind Time 1.02s (± 0.40%) 1.01s (± 0.80%) ~ 1.00s 1.02s p=0.248 n=6
Check Time 6.21s (± 0.38%) 6.19s (± 0.37%) ~ 6.16s 6.22s p=0.419 n=6
Emit Time 3.47s (± 0.42%) 3.39s (± 0.43%) -0.08s (- 2.21%) 3.38s 3.41s p=0.005 n=6
Total Time 12.73s (± 0.21%) 12.63s (± 0.22%) -0.10s (- 0.79%) 12.60s 12.68s p=0.005 n=6
material-ui - node (v18.15.0, x64)
Memory used 480,288k (± 0.01%) 479,995k (± 0.00%) -293k (- 0.06%) 479,966k 480,016k p=0.005 n=6
Parse Time 2.97s (± 0.89%) 3.00s (± 0.62%) ~ 2.98s 3.03s p=0.211 n=6
Bind Time 0.86s (± 0.73%) 0.85s (± 0.60%) ~ 0.85s 0.86s p=0.091 n=6
Check Time 16.67s (± 0.18%) 16.70s (± 0.59%) ~ 16.52s 16.77s p=0.227 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.50s (± 0.14%) 20.55s (± 0.39%) ~ 20.41s 20.63s p=0.172 n=6
xstate - node (v18.15.0, x64)
Memory used 545,649k (± 0.01%) 544,670k (± 0.01%) -979k (- 0.18%) 544,606k 544,760k p=0.005 n=6
Parse Time 3.51s (± 0.29%) 3.48s (± 0.16%) -0.03s (- 0.81%) 3.48s 3.49s p=0.004 n=6
Bind Time 1.49s (± 0.42%) 1.46s (± 0.75%) -0.03s (- 2.01%) 1.44s 1.47s p=0.004 n=6
Check Time 2.81s (± 1.10%) 2.79s (± 0.92%) ~ 2.75s 2.81s p=0.293 n=6
Emit Time 0.07s (± 7.03%) 0.08s (± 6.19%) +0.01s (+13.64%) 0.08s 0.09s p=0.018 n=6
Total Time 7.89s (± 0.58%) 7.81s (± 0.39%) -0.08s (- 0.99%) 7.76s 7.85s p=0.010 n=6
Angular - node (v20.5.1, x64)
Memory used 312,752k (± 0.00%) 312,128k (± 0.01%) -624k (- 0.20%) 312,108k 312,206k p=0.005 n=6
Parse Time 3.03s (± 0.28%) 3.05s (± 0.91%) ~ 3.00s 3.07s p=0.100 n=6
Bind Time 0.90s (± 0.58%) 0.90s (± 0.00%) ~ 0.90s 0.90s p=0.174 n=6
Check Time 7.77s (± 0.13%) 7.72s (± 0.30%) -0.05s (- 0.71%) 7.70s 7.76s p=0.006 n=6
Emit Time 7.35s (± 0.16%) 7.26s (± 0.11%) -0.09s (- 1.22%) 7.25s 7.27s p=0.005 n=6
Total Time 19.06s (± 0.09%) 18.93s (± 0.19%) -0.12s (- 0.65%) 18.87s 18.98s p=0.004 n=6
Compiler-Unions - node (v20.5.1, x64)
Memory used 192,179k (± 1.24%) 191,740k (± 1.24%) ~ 190,331k 196,151k p=0.173 n=6
Parse Time 1.49s (± 2.47%) 1.50s (± 2.21%) ~ 1.45s 1.54s p=0.743 n=6
Bind Time 0.77s (± 1.42%) 0.77s (± 1.16%) ~ 0.76s 0.78s p=1.000 n=6
Check Time 8.68s (± 0.49%) 8.60s (± 1.07%) ~ 8.50s 8.73s p=0.173 n=6
Emit Time 2.61s (± 0.70%) 2.60s (± 0.46%) ~ 2.59s 2.62s p=0.742 n=6
Total Time 13.54s (± 0.58%) 13.48s (± 0.58%) ~ 13.42s 13.58s p=0.170 n=6
Monaco - node (v20.5.1, x64)
Memory used 352,267k (± 0.01%) 351,893k (± 0.01%) -375k (- 0.11%) 351,866k 351,935k p=0.005 n=6
Parse Time 2.64s (± 0.98%) 2.64s (± 0.73%) ~ 2.62s 2.67s p=0.622 n=6
Bind Time 0.96s (± 1.69%) 0.97s (± 0.77%) ~ 0.96s 0.98s p=0.454 n=6
Check Time 6.79s (± 0.29%) 6.78s (± 0.22%) ~ 6.77s 6.81s p=0.618 n=6
Emit Time 3.88s (± 0.34%) 3.83s (± 0.27%) -0.05s (- 1.25%) 3.82s 3.85s p=0.004 n=6
Total Time 14.27s (± 0.31%) 14.23s (± 0.29%) ~ 14.18s 14.29s p=0.108 n=6
TFS - node (v20.5.1, x64)
Memory used 301,751k (± 0.00%) 301,337k (± 0.00%) -415k (- 0.14%) 301,318k 301,357k p=0.005 n=6
Parse Time 2.05s (± 0.65%) 2.05s (± 0.41%) ~ 2.05s 2.07s p=0.211 n=6
Bind Time 1.06s (± 0.77%) 1.04s (± 0.49%) -0.01s (- 1.26%) 1.04s 1.05s p=0.014 n=6
Check Time 6.04s (± 0.39%) 6.03s (± 0.20%) ~ 6.01s 6.04s p=0.373 n=6
Emit Time 3.75s (± 0.52%) 3.71s (± 0.45%) -0.04s (- 1.11%) 3.69s 3.73s p=0.008 n=6
Total Time 12.90s (± 0.27%) 12.83s (± 0.11%) -0.06s (- 0.49%) 12.81s 12.85s p=0.012 n=6
material-ui - node (v20.5.1, x64)
Memory used 485,121k (± 0.01%) 484,811k (± 0.00%) -310k (- 0.06%) 484,795k 484,851k p=0.005 n=6
Parse Time 3.08s (± 0.36%) 3.07s (± 0.17%) ~ 3.07s 3.08s p=0.247 n=6
Bind Time 1.05s (± 1.11%) 1.04s (± 0.50%) -0.01s (- 1.11%) 1.03s 1.04s p=0.039 n=6
Check Time 15.85s (± 0.47%) 15.85s (± 0.38%) ~ 15.79s 15.96s p=1.000 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 19.99s (± 0.28%) 19.97s (± 0.34%) ~ 19.90s 20.09s p=0.520 n=6
xstate - node (v20.5.1, x64)
Memory used 544,703k (± 0.02%) 543,883k (± 0.01%) -819k (- 0.15%) 543,834k 543,943k p=0.005 n=6
Parse Time 3.72s (± 0.40%) 3.74s (± 0.15%) ~ 3.73s 3.74s p=0.091 n=6
Bind Time 1.54s (± 0.49%) 1.54s (± 0.71%) ~ 1.52s 1.55s p=0.604 n=6
Check Time 2.85s (± 0.79%) 2.83s (± 0.52%) ~ 2.82s 2.85s p=0.191 n=6
Emit Time 0.15s (± 9.31%) 0.15s (± 0.00%) ~ 0.15s 0.15s p=1.000 n=6
Total Time 8.25s (± 0.36%) 8.25s (± 0.34%) ~ 8.22s 8.29s p=0.808 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
  • node (v18.15.0, x64)
  • node (v20.5.1, x64)
Scenarios
  • Angular - node (v16.17.1, x64)
  • Compiler-Unions - node (v16.17.1, x64)
  • Monaco - node (v16.17.1, x64)
  • TFS - node (v16.17.1, x64)
  • material-ui - node (v16.17.1, x64)
  • xstate - node (v16.17.1, x64)
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
  • Angular - node (v20.5.1, x64)
  • Compiler-Unions - node (v20.5.1, x64)
  • Monaco - node (v20.5.1, x64)
  • TFS - node (v20.5.1, x64)
  • material-ui - node (v20.5.1, x64)
  • xstate - node (v20.5.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,494ms (± 0.11%) 2,482ms (± 0.23%) -12ms (- 0.48%) 2,474ms 2,487ms p=0.005 n=6
Req 2 - geterr 5,920ms (± 0.30%) 5,846ms (± 0.53%) -74ms (- 1.24%) 5,801ms 5,894ms p=0.008 n=6
Req 3 - references 342ms (± 0.32%) 350ms (± 0.35%) +8ms (+ 2.24%) 348ms 351ms p=0.004 n=6
Req 4 - navto 277ms (± 0.64%) 263ms (± 0.71%) 🟩-13ms (- 4.76%) 262ms 267ms p=0.004 n=6
Req 5 - completionInfo count 1,356 (± 0.00%) 1,356 (± 0.00%) ~ 1,356 1,356 p=1.000 n=6
Req 5 - completionInfo 82ms (± 8.00%) 81ms (± 9.70%) ~ 75ms 92ms p=0.368 n=6
CompilerTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,601ms (± 0.60%) 2,585ms (± 0.17%) ~ 2,578ms 2,589ms p=0.126 n=6
Req 2 - geterr 4,718ms (± 0.15%) 4,687ms (± 0.14%) -32ms (- 0.67%) 4,678ms 4,694ms p=0.005 n=6
Req 3 - references 350ms (± 0.21%) 355ms (± 0.11%) +5ms (+ 1.52%) 355ms 356ms p=0.003 n=6
Req 4 - navto 270ms (± 0.38%) 273ms (± 0.50%) +4ms (+ 1.36%) 271ms 275ms p=0.006 n=6
Req 5 - completionInfo count 1,518 (± 0.00%) 1,518 (± 0.00%) ~ 1,518 1,518 p=1.000 n=6
Req 5 - completionInfo 79ms (± 0.80%) 74ms (± 3.47%) 🟩-5ms (- 6.54%) 72ms 79ms p=0.014 n=6
xstateTSServer - node (v16.17.1, x64)
Req 1 - updateOpen 2,715ms (± 0.11%) 2,695ms (± 0.51%) ~ 2,676ms 2,719ms p=0.065 n=6
Req 2 - geterr 1,956ms (± 0.66%) 1,910ms (± 2.71%) ~ 1,844ms 1,962ms p=0.149 n=6
Req 3 - references 122ms (± 9.23%) 121ms (± 9.52%) ~ 110ms 133ms p=0.377 n=6
Req 4 - navto 353ms (± 0.29%) 372ms (± 1.13%) +19ms (+ 5.43%) 368ms 376ms p=0.004 n=6
Req 5 - completionInfo count 2,071 (± 0.00%) 2,071 (± 0.00%) ~ 2,071 2,071 p=1.000 n=6
Req 5 - completionInfo 313ms (± 0.98%) 321ms (± 2.08%) +8ms (+ 2.56%) 314ms 329ms p=0.036 n=6
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • CompilerTSServer - node (v16.17.1, x64)
  • Compiler-UnionsTSServer - node (v16.17.1, x64)
  • xstateTSServer - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v16.17.1, x64)
Execution time 155.68ms (± 0.21%) 154.04ms (± 0.17%) -1.64ms (- 1.05%) 153.00ms 158.31ms p=0.000 n=600
tsserver-startup - node (v16.17.1, x64)
Execution time 230.82ms (± 0.15%) 227.79ms (± 0.12%) -3.03ms (- 1.31%) 226.63ms 230.58ms p=0.000 n=600
tsserverlibrary-startup - node (v16.17.1, x64)
Execution time 234.74ms (± 0.11%) 232.76ms (± 0.14%) -1.97ms (- 0.84%) 230.75ms 236.41ms p=0.000 n=600
typescript-startup - node (v16.17.1, x64)
Execution time 218.13ms (± 0.14%) 214.73ms (± 0.13%) -3.41ms (- 1.56%) 213.72ms 218.05ms p=0.000 n=600
System info unknown
Hosts
  • node (v16.17.1, x64)
Scenarios
  • tsc-startup - node (v16.17.1, x64)
  • tsserver-startup - node (v16.17.1, x64)
  • tsserverlibrary-startup - node (v16.17.1, x64)
  • typescript-startup - node (v16.17.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey

This comment was marked as outdated.

Sign in to view
@typescript-bot

This comment was marked as outdated.

Sign in to view
@typescript-bot

This comment was marked as outdated.

Sign in to view
@jakebailey

This comment was marked as outdated.

Sign in to view
@typescript-bot

This comment was marked as outdated.

Sign in to view
@jakebailey
Copy link
Member Author

@typescript-bot new perf test bun

@typescript-bot
Copy link
Collaborator

typescript-bot commented Aug 30, 2023 •
edited

Heya @jakebailey, I've started to run the bun perf test suite on this PR at e67fbc0. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

Compiler

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - bun (v0.7.3, x64)
Memory used 321,810k (± 0.23%) 322,091k (± 0.28%) ~ 321,114k 323,375k p=0.689 n=6
Parse Time 2.32s (± 0.72%) 2.35s (± 0.79%) +0.03s (+ 1.44%) 2.33s 2.38s p=0.012 n=6
Bind Time 0.95s (± 5.36%) 0.94s (± 5.06%) ~ 0.90s 1.01s p=0.687 n=6
Check Time 7.94s (± 1.68%) 8.04s (± 0.37%) ~ 8.00s 8.07s p=0.254 n=6
Emit Time 6.44s (± 0.42%) 6.47s (± 0.77%) ~ 6.42s 6.55s p=0.466 n=6
Total Time 17.65s (± 0.56%) 17.80s (± 0.38%) +0.15s (+ 0.84%) 17.73s 17.90s p=0.029 n=6
Compiler-Unions - bun (v0.7.3, x64)
Memory used 246,169k (± 3.04%) 243,993k (± 2.10%) ~ 237,438k 249,524k p=0.575 n=6
Parse Time 1.26s (± 0.71%) 1.28s (± 1.74%) ~ 1.25s 1.31s p=0.164 n=6
Bind Time 0.85s (± 2.53%) 0.85s (± 0.96%) ~ 0.84s 0.86s p=0.682 n=6
Check Time 8.90s (± 0.23%) 9.00s (± 0.71%) ~ 8.88s 9.06s p=0.053 n=6
Emit Time 2.76s (± 0.86%) 2.74s (± 0.64%) ~ 2.72s 2.77s p=0.072 n=6
Total Time 13.78s (± 0.29%) 13.86s (± 0.35%) +0.09s (+ 0.63%) 13.77s 13.91s p=0.015 n=6
Monaco - bun (v0.7.3, x64)
Memory used 370,241k (± 0.37%) 370,192k (± 0.28%) ~ 369,196k 371,592k p=1.000 n=6
Parse Time 2.06s (± 0.57%) 2.05s (± 0.85%) ~ 2.03s 2.08s p=0.084 n=6
Bind Time 1.11s (± 2.48%) 1.11s (± 2.26%) ~ 1.08s 1.14s p=0.807 n=6
Check Time 7.19s (± 0.65%) 7.21s (± 0.52%) ~ 7.17s 7.28s p=0.746 n=6
Emit Time 3.75s (± 1.08%) 3.76s (± 0.93%) ~ 3.73s 3.82s p=0.571 n=6
Total Time 14.11s (± 0.29%) 14.13s (± 0.51%) ~ 14.06s 14.26s p=0.936 n=6
TFS - bun (v0.7.3, x64)
Memory used 317,835k (± 0.13%) 317,724k (± 0.25%) ~ 316,647k 319,042k p=0.378 n=6
Parse Time 1.80s (± 0.67%) 1.79s (± 0.55%) ~ 1.78s 1.80s p=0.138 n=6
Bind Time 1.00s (± 1.21%) 0.98s (± 1.78%) ~ 0.96s 1.01s p=0.059 n=6
Check Time 6.62s (± 0.64%) 6.68s (± 0.54%) +0.05s (+ 0.81%) 6.62s 6.73s p=0.045 n=6
Emit Time 3.44s (± 0.83%) 3.50s (± 1.23%) +0.06s (+ 1.79%) 3.45s 3.58s p=0.013 n=6
Total Time 12.88s (± 0.38%) 12.96s (± 0.53%) ~ 12.87s 13.05s p=0.065 n=6
material-ui - bun (v0.7.3, x64)
Memory used 537,607k (± 0.25%) 508,460k (± 7.12%) ~ 464,102k 543,543k p=0.471 n=6
Parse Time 2.50s (± 0.75%) 2.45s (± 0.71%) -0.04s (- 1.67%) 2.44s 2.48s p=0.007 n=6
Bind Time 0.71s (± 1.06%) 0.76s (± 3.70%) +0.05s (+ 6.82%) 0.73s 0.80s p=0.005 n=6
Check Time 15.76s (± 0.53%) 15.80s (± 0.77%) ~ 15.66s 15.99s p=0.423 n=6
Emit Time 0.00s (±244.70%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=0.405 n=6
Total Time 18.97s (± 0.41%) 19.02s (± 0.62%) ~ 18.88s 19.19s p=0.374 n=6
xstate - bun (v0.7.3, x64)
Memory used 471,655k (± 9.65%) 472,130k (± 8.15%) ~ 436,287k 509,641k p=1.000 n=6
Parse Time 3.60s (± 1.11%) 3.58s (± 0.38%) ~ 3.56s 3.60s p=0.627 n=6
Bind Time 1.41s (± 2.84%) 1.41s (± 1.17%) ~ 1.39s 1.43s p=0.870 n=6
Check Time 3.54s (± 0.81%) 3.62s (± 0.33%) +0.08s (+ 2.31%) 3.60s 3.63s p=0.004 n=6
Emit Time 0.22s (± 9.08%) 0.21s (± 3.55%) ~ 0.20s 0.22s p=0.652 n=6
Total Time 8.77s (± 0.51%) 8.83s (± 0.30%) +0.06s (+ 0.74%) 8.80s 8.86s p=0.029 n=6
System info unknown
Hosts
  • bun (v0.7.3, x64)
Scenarios
  • Angular - bun (v0.7.3, x64)
  • Compiler-Unions - bun (v0.7.3, x64)
  • Monaco - bun (v0.7.3, x64)
  • TFS - bun (v0.7.3, x64)
  • material-ui - bun (v0.7.3, x64)
  • xstate - bun (v0.7.3, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - bun (v0.7.3, x64)
Execution time 425.30ms (± 0.10%) 423.55ms (± 0.11%) -1.75ms (- 0.41%) 422.65ms 432.93ms p=0.000 n=600
tsserverlibrary-startup - bun (v0.7.3, x64)
Execution time 661.41ms (± 0.14%) 656.76ms (± 0.13%) -4.65ms (- 0.70%) 655.22ms 678.21ms p=0.000 n=600
typescript-startup - bun (v0.7.3, x64)
Execution time 662.40ms (± 0.11%) 654.86ms (± 0.14%) -7.54ms (- 1.14%) 653.45ms 672.29ms p=0.000 n=600
System info unknown
Hosts
  • bun (v0.7.3, x64)
Scenarios
  • tsc-startup - bun (v0.7.3, x64)
  • tsserverlibrary-startup - bun (v0.7.3, x64)
  • typescript-startup - bun (v0.7.3, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey jakebailey mentioned this pull request Aug 30, 2023
Closed
@jakebailey
Copy link
Member Author

@typescript-bot perf test public

@typescript-bot
Copy link
Collaborator

typescript-bot commented Jan 11, 2024 •
edited

Heya @jakebailey, I've started to run the public perf test suite on this PR at 0c249d9. You can monitor the build here.

Update: The results are in!

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
mui-docs - node (v20.5.1, x64)
Memory used 1,735,248k (± 0.00%) 1,734,982k (± 0.00%) -266k (- 0.02%) 1,734,964k 1,735,005k p=0.005 n=6
Parse Time 6.83s (± 0.46%) 6.81s (± 0.44%) ~ 6.77s 6.85s p=0.333 n=6
Bind Time 2.42s (± 1.10%) 2.42s (± 0.56%) ~ 2.41s 2.44s p=0.459 n=6
Check Time 52.39s (± 0.40%) 52.55s (± 0.60%) ~ 51.97s 52.91s p=0.128 n=6
Emit Time 0.15s (± 0.00%) 0.15s (± 0.00%) ~ 0.15s 0.15s p=1.000 n=6
Total Time 61.78s (± 0.30%) 61.94s (± 0.51%) ~ 61.35s 62.28s p=0.149 n=6
self-build-src - node (v20.5.1, x64)
Memory used 2,751,144k (± 6.91%) 2,571,396k (± 0.03%) 🟩-179,748k (- 6.53%) 2,570,753k 2,573,098k p=0.005 n=6
Parse Time 5.05s (± 0.81%) 5.04s (± 0.58%) ~ 4.99s 5.07s p=0.748 n=6
Bind Time 1.99s (± 0.49%) 2.03s (± 1.35%) +0.04s (+ 1.84%) 1.99s 2.07s p=0.016 n=6
Check Time 32.12s (± 0.35%) 32.20s (± 0.16%) ~ 32.12s 32.26s p=0.298 n=6
Emit Time 2.79s (± 3.97%) 2.79s (± 4.04%) ~ 2.62s 2.94s p=0.936 n=6
Total Time 41.97s (± 0.22%) 42.07s (± 0.19%) ~ 41.97s 42.17s p=0.093 n=6
self-compiler - node (v20.5.1, x64)
Memory used 419,228k (± 0.01%) 419,355k (± 0.29%) ~ 418,746k 421,793k p=0.066 n=6
Parse Time 2.88s (± 0.48%) 2.87s (± 0.97%) ~ 2.84s 2.91s p=1.000 n=6
Bind Time 1.13s (± 0.74%) 1.14s (± 0.36%) ~ 1.13s 1.14s p=0.285 n=6
Check Time 14.12s (± 0.33%) 14.10s (± 0.41%) ~ 14.03s 14.17s p=0.688 n=6
Emit Time 1.05s (± 1.17%) 1.04s (± 0.72%) ~ 1.03s 1.05s p=1.000 n=6
Total Time 19.17s (± 0.33%) 19.15s (± 0.28%) ~ 19.08s 19.22s p=0.467 n=6
vscode - node (v20.5.1, x64)
Memory used 2,828,007k (± 0.01%) 2,827,290k (± 0.01%) -717k (- 0.03%) 2,827,184k 2,827,570k p=0.005 n=6
Parse Time 10.72s (± 0.17%) 10.71s (± 0.22%) ~ 10.67s 10.73s p=0.324 n=6
Bind Time 3.41s (± 0.82%) 3.45s (± 0.15%) +0.04s (+ 1.22%) 3.45s 3.46s p=0.010 n=6
Check Time 56.23s (± 0.45%) 56.64s (± 0.41%) +0.41s (+ 0.73%) 56.25s 56.88s p=0.031 n=6
Emit Time 16.81s (± 8.95%) 19.10s (± 7.87%) ~ 16.03s 19.77s p=0.297 n=6
Total Time 87.17s (± 1.84%) 89.90s (± 1.88%) ~ 86.47s 90.79s p=0.066 n=6
webpack - node (v20.5.1, x64)
Memory used 390,786k (± 0.01%) 390,445k (± 0.01%) -341k (- 0.09%) 390,395k 390,505k p=0.005 n=6
Parse Time 3.32s (± 0.40%) 3.30s (± 0.49%) ~ 3.28s 3.32s p=0.067 n=6
Bind Time 1.43s (± 0.82%) 1.44s (± 0.84%) ~ 1.42s 1.45s p=0.323 n=6
Check Time 12.78s (± 0.21%) 12.70s (± 0.31%) -0.08s (- 0.66%) 12.64s 12.74s p=0.005 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 17.54s (± 0.13%) 17.44s (± 0.18%) -0.10s (- 0.57%) 17.40s 17.47s p=0.005 n=6
System info unknown
Hosts
  • node (v20.5.1, x64)
Scenarios
  • mui-docs - node (v20.5.1, x64)
  • self-build-src - node (v20.5.1, x64)
  • self-compiler - node (v20.5.1, x64)
  • vscode - node (v20.5.1, x64)
  • webpack - node (v20.5.1, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Mar 19, 2024 •
edited

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,740k (± 0.01%) 295,106k (± 0.00%) -633k (- 0.21%) 295,088k 295,124k p=0.005 n=6
Parse Time 2.66s (± 0.19%) 2.62s (± 0.24%) -0.04s (- 1.63%) 2.61s 2.63s p=0.004 n=6
Bind Time 0.83s (± 0.66%) 0.83s (± 0.62%) +0.01s (+ 1.01%) 0.83s 0.84s p=0.038 n=6
Check Time 8.21s (± 0.30%) 8.06s (± 0.38%) -0.15s (- 1.79%) 8.01s 8.09s p=0.005 n=6
Emit Time 7.13s (± 0.71%) 6.98s (± 0.47%) -0.16s (- 2.20%) 6.95s 7.03s p=0.005 n=6
Total Time 18.83s (± 0.31%) 18.50s (± 0.30%) -0.33s (- 1.77%) 18.41s 18.57s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,791k (± 0.99%) 191,565k (± 0.06%) -2,226k (- 1.15%) 191,485k 191,801k p=0.005 n=6
Parse Time 1.36s (± 1.23%) 1.36s (± 2.25%) ~ 1.30s 1.38s p=0.681 n=6
Bind Time 0.72s (± 0.57%) 0.72s (± 0.57%) ~ 0.72s 0.73s p=1.000 n=6
Check Time 9.51s (± 0.82%) 9.31s (± 0.57%) -0.20s (- 2.10%) 9.24s 9.38s p=0.005 n=6
Emit Time 2.66s (± 0.46%) 2.64s (± 0.41%) ~ 2.63s 2.66s p=0.062 n=6
Total Time 14.24s (± 0.53%) 14.03s (± 0.58%) -0.21s (- 1.50%) 13.90s 14.13s p=0.005 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,366k (± 0.00%) 347,008k (± 0.01%) -358k (- 0.10%) 346,990k 347,042k p=0.005 n=6
Parse Time 2.48s (± 0.21%) 2.48s (± 0.59%) ~ 2.46s 2.50s p=0.451 n=6
Bind Time 0.93s (± 0.44%) 0.93s (± 0.44%) ~ 0.93s 0.94s p=0.218 n=6
Check Time 7.02s (± 0.28%) 7.04s (± 0.47%) ~ 7.01s 7.10s p=0.413 n=6
Emit Time 4.07s (± 0.34%) 4.06s (± 0.55%) ~ 4.03s 4.08s p=1.000 n=6
Total Time 14.49s (± 0.24%) 14.51s (± 0.26%) ~ 14.44s 14.54s p=0.332 n=6
TFS - node (v18.15.0, x64)
Memory used 302,727k (± 0.00%) 302,256k (± 0.01%) -471k (- 0.16%) 302,214k 302,312k p=0.005 n=6
Parse Time 2.01s (± 1.02%) 2.00s (± 0.55%) ~ 1.98s 2.01s p=0.503 n=6
Bind Time 1.00s (± 0.98%) 1.00s (± 0.82%) ~ 0.99s 1.01s p=0.394 n=6
Check Time 6.32s (± 0.60%) 6.30s (± 0.33%) ~ 6.27s 6.33s p=0.258 n=6
Emit Time 3.60s (± 0.48%) 3.54s (± 0.52%) -0.06s (- 1.53%) 3.52s 3.57s p=0.006 n=6
Total Time 12.93s (± 0.41%) 12.84s (± 0.17%) -0.09s (- 0.71%) 12.80s 12.86s p=0.008 n=6
material-ui - node (v18.15.0, x64)
Memory used 509,922k (± 0.00%) 509,549k (± 0.00%) -373k (- 0.07%) 509,531k 509,568k p=0.005 n=6
Parse Time 2.66s (± 0.41%) 2.70s (± 0.20%) +0.04s (+ 1.32%) 2.69s 2.70s p=0.004 n=6
Bind Time 0.98s (± 1.06%) 0.98s (± 1.00%) ~ 0.97s 0.99s p=0.673 n=6
Check Time 17.23s (± 0.39%) 17.26s (± 0.27%) ~ 17.18s 17.30s p=0.336 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 20.88s (± 0.38%) 20.93s (± 0.23%) ~ 20.86s 20.99s p=0.198 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,718,974k (± 0.00%) 1,718,497k (± 0.00%) -477k (- 0.03%) 1,718,463k 1,718,524k p=0.005 n=6
Parse Time 6.51s (± 0.40%) 6.58s (± 0.53%) +0.07s (+ 1.08%) 6.56s 6.65s p=0.005 n=6
Bind Time 2.36s (± 0.27%) 2.34s (± 0.66%) -0.02s (- 0.85%) 2.32s 2.36s p=0.024 n=6
Check Time 56.12s (± 0.36%) 56.40s (± 0.56%) ~ 55.92s 56.77s p=0.128 n=6
Emit Time 0.13s (± 0.00%) 0.14s (± 4.05%) ~ 0.13s 0.14s p=0.071 n=6
Total Time 65.12s (± 0.33%) 65.45s (± 0.53%) ~ 64.94s 65.89s p=0.093 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,394,975k (± 0.02%) 2,389,428k (± 0.02%) -5,547k (- 0.23%) 2,388,869k 2,390,500k p=0.005 n=6
Parse Time 4.98s (± 1.40%) 5.06s (± 1.19%) +0.08s (+ 1.64%) 4.98s 5.12s p=0.037 n=6
Bind Time 1.91s (± 1.32%) 1.88s (± 1.34%) ~ 1.85s 1.92s p=0.063 n=6
Check Time 33.57s (± 0.32%) 33.36s (± 0.21%) -0.21s (- 0.64%) 33.27s 33.45s p=0.013 n=6
Emit Time 2.71s (± 1.16%) 2.61s (± 1.03%) 🟩-0.10s (- 3.57%) 2.57s 2.64s p=0.005 n=6
Total Time 43.18s (± 0.38%) 42.93s (± 0.26%) -0.25s (- 0.58%) 42.73s 43.04s p=0.013 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,056k (± 0.01%) 415,592k (± 0.01%) -464k (- 0.11%) 415,533k 415,665k p=0.005 n=6
Parse Time 2.83s (± 0.68%) 2.74s (± 2.71%) 🟩-0.09s (- 3.06%) 2.64s 2.81s p=0.005 n=6
Bind Time 1.07s (± 0.48%) 1.11s (± 5.71%) ~ 1.07s 1.22s p=0.070 n=6
Check Time 15.33s (± 0.50%) 15.19s (± 0.39%) -0.14s (- 0.91%) 15.13s 15.25s p=0.010 n=6
Emit Time 1.14s (± 0.92%) 1.11s (± 0.57%) 🟩-0.03s (- 3.06%) 1.10s 1.12s p=0.004 n=6
Total Time 20.37s (± 0.38%) 20.15s (± 0.32%) -0.22s (- 1.10%) 20.07s 20.22s p=0.005 n=6
vscode - node (v18.15.0, x64)
Memory used 2,884,074k (± 0.00%) 2,883,374k (± 0.00%) -700k (- 0.02%) 2,883,258k 2,883,414k p=0.005 n=6
Parse Time 10.81s (± 0.26%) 10.78s (± 0.32%) ~ 10.74s 10.82s p=0.212 n=6
Bind Time 3.47s (± 0.30%) 3.48s (± 0.56%) ~ 3.46s 3.51s p=0.263 n=6
Check Time 61.41s (± 0.47%) 61.47s (± 0.34%) ~ 61.09s 61.69s p=0.872 n=6
Emit Time 16.44s (± 0.32%) 16.22s (± 0.49%) -0.22s (- 1.35%) 16.13s 16.34s p=0.005 n=6
Total Time 92.12s (± 0.30%) 91.94s (± 0.30%) ~ 91.47s 92.28s p=0.378 n=6
webpack - node (v18.15.0, x64)
Memory used 408,028k (± 0.01%) 407,754k (± 0.01%) -274k (- 0.07%) 407,690k 407,820k p=0.005 n=6
Parse Time 3.23s (± 0.23%) 3.20s (± 0.25%) -0.03s (- 0.88%) 3.19s 3.21s p=0.004 n=6
Bind Time 1.38s (± 0.30%) 1.40s (± 0.74%) +0.02s (+ 1.57%) 1.39s 1.42s p=0.004 n=6
Check Time 14.27s (± 0.20%) 14.23s (± 0.35%) ~ 14.19s 14.30s p=0.169 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 18.88s (± 0.19%) 18.84s (± 0.25%) ~ 18.79s 18.91s p=0.198 n=6
xstate - node (v18.15.0, x64)
Memory used 513,080k (± 0.01%) 512,180k (± 0.01%) -900k (- 0.18%) 512,089k 512,258k p=0.005 n=6
Parse Time 3.28s (± 0.37%) 3.25s (± 0.45%) -0.03s (- 0.81%) 3.23s 3.27s p=0.018 n=6
Bind Time 1.54s (± 0.35%) 1.55s (± 0.26%) ~ 1.55s 1.56s p=0.054 n=6
Check Time 2.86s (± 0.29%) 2.88s (± 0.36%) +0.03s (+ 0.99%) 2.87s 2.90s p=0.005 n=6
Emit Time 0.07s (± 7.03%) 0.07s (± 7.03%) ~ 0.07s 0.08s p=1.000 n=6
Total Time 7.75s (± 0.33%) 7.76s (± 0.16%) ~ 7.75s 7.78s p=0.518 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,336ms (± 0.78%) 2,337ms (± 0.78%) ~ 2,315ms 2,359ms p=1.000 n=6
Req 2 - geterr 5,794ms (± 0.37%) 5,677ms (± 0.53%) -117ms (- 2.01%) 5,656ms 5,738ms p=0.005 n=6
Req 3 - references 325ms (± 1.04%) 332ms (± 0.82%) +7ms (+ 2.00%) 330ms 337ms p=0.032 n=6
Req 4 - navto 272ms (± 0.19%) 265ms (± 0.15%) -7ms (- 2.63%) 265ms 266ms p=0.003 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 95ms (± 0.43%) 94ms (± 1.16%) ~ 93ms 96ms p=0.090 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,499ms (± 0.89%) 2,469ms (± 0.53%) -30ms (- 1.19%) 2,443ms 2,480ms p=0.016 n=6
Req 2 - geterr 4,264ms (± 0.33%) 4,159ms (± 0.49%) -105ms (- 2.45%) 4,132ms 4,184ms p=0.005 n=6
Req 3 - references 332ms (± 0.12%) 335ms (± 0.22%) +3ms (+ 0.90%) 334ms 336ms p=0.003 n=6
Req 4 - navto 283ms (± 0.45%) 278ms (± 0.44%) -6ms (- 1.94%) 276ms 279ms p=0.004 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 78ms (± 0.81%) 85ms (± 6.80%) 🔻+7ms (+ 8.33%) 78ms 90ms p=0.020 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,618ms (± 0.11%) 2,590ms (± 0.31%) -28ms (- 1.06%) 2,579ms 2,598ms p=0.005 n=6
Req 2 - geterr 1,725ms (± 1.98%) 1,720ms (± 1.64%) ~ 1,682ms 1,759ms p=0.689 n=6
Req 3 - references 115ms (±10.05%) 119ms (± 6.94%) ~ 102ms 123ms p=0.683 n=6
Req 4 - navto 371ms (± 0.20%) 375ms (± 0.90%) +4ms (+ 0.99%) 372ms 380ms p=0.011 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 308ms (± 1.42%) 308ms (± 1.41%) ~ 301ms 313ms p=0.517 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 153.76ms (± 0.17%) 151.55ms (± 0.17%) -2.20ms (- 1.43%) 150.52ms 154.90ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 228.85ms (± 0.16%) 229.69ms (± 0.19%) +0.84ms (+ 0.37%) 228.13ms 236.57ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 222.61ms (± 0.15%) 220.92ms (± 0.19%) -1.70ms (- 0.76%) 219.44ms 227.40ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 223.53ms (± 0.15%) 222.11ms (± 0.20%) -1.42ms (- 0.64%) 220.38ms 228.18ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@7nik 7nik mentioned this pull request Apr 23, 2024
Merged
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jakebailey jakebailey

Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
10 participants
@jakebailey @typescript-bot @Andarist @fatcerberus @Jarred-Sumner @marvinhagemeister @robpalme @mprobst @Jack-Works @syg