chore: speed up local dev #686

eh-am · 2022-01-08T01:05:50Z

maxWorkers=1 as per Speed up performance massively with --maxWorkers=1 kulshekhar/ts-jest#1044
a scoped includes to only care about the webapp
a tsconfig for tests where we manually set which types are required
different eslint rules for tests
use esbuild instead of babel
use webpack-livereload-plugin to add a livereload server to the html files served by go (removing the need for the double server shenanigans)
in local dev only build the index.html since that's the most common case (see webapp/README.md for more info)

Local build

Before

21:32:05 webpack | 0 (webpack 5.65.0) compiled with 1 warning in 3296 ms

After

21:33:46 webpack | webpack 5.65.0 compiled with 1 warning in 446 ms

Tests

Before

Test Suites: 15 passed, 15 total
Tests:       193 passed, 193 total
Snapshots:   4 passed, 4 total
Time:        25.552 s
Ran all test suites.
Done in 26.96s.

After

Test Suites: 15 passed, 15 total
Tests:       193 passed, 193 total
Snapshots:   4 passed, 4 total
Time:        17.276 s
Ran all test suites.
Done in 18.88s.

But the biggest improvement seems to be when running tests with --watch.

Before

Test Suites: 1 failed, 1 total
Tests:       3 failed, 3 passed, 6 total
Snapshots:   0 total
Time:        8.744 s, estimated 10 s
Ran all test suites matching /Sidebar.spec.tsx/i.

After

Test Suites: 1 failed, 1 total
Tests:       3 failed, 3 passed, 6 total
Snapshots:   0 total
Time:        1.753 s, estimated 2 s
Ran all test suites matching /Sidebar.spec.tsx/i.

Prod Build

Before

$ time yarn build 
(...) 
Done in 21.36s.

________________________________________________________
Executed in   21.53 secs    fish           external
   usr time   38.73 secs    0.51 millis   38.73 secs
   sys time    0.95 secs    1.05 millis    0.95 secs

After

$ time yarn build  
(...)
Done in 11.45s.

________________________________________________________
Executed in   11.61 secs    fish           external
   usr time   19.26 secs  811.00 micros   19.26 secs
   sys time    0.66 secs  909.00 micros    0.66 secs

as per kulshekhar/ts-jest#259 (comment) using maxWorkers: 1 ends up being faster

github-actions · 2022-01-08T01:08:50Z

size-limit report 📦

Path	Size	Loading time (3g)	Running time (snapdragon)	Total time
webapp/public/assets/app.js	323.18 KB (+26.94% 🔺)	6.5 s (+26.94% 🔺)	884 ms (-1.12% 🔽)	7.4 s

codecov · 2022-01-08T01:12:24Z

Codecov Report

Merging #686 (0855729) into main (ad4722d) will increase coverage by 0.38%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #686      +/-   ##
==========================================
+ Coverage   75.45%   75.83%   +0.38%     
==========================================
  Files          47       47              
  Lines        1617     1758     +141     
  Branches      295      326      +31     
==========================================
+ Hits         1220     1333     +113     
- Misses        368      397      +29     
+ Partials       29       28       -1

Impacted Files	Coverage Δ
webapp/javascript/ui/Notifications.tsx	`45.17% <0.00%> (-14.83%)`	⬇️
...nts/FlameGraph/FlameGraphComponent/ContextMenu.tsx	`93.03% <0.00%> (-3.64%)`	⬇️
webapp/javascript/ui/Sidebar.tsx	`93.03% <0.00%> (-3.40%)`	⬇️
...omponents/FlameGraph/FlameGraphComponent/index.tsx	`84.05% <0.00%> (-1.01%)`	⬇️
webapp/javascript/components/Sidebar.tsx	`84.06% <0.00%> (-0.84%)`	⬇️
webapp/javascript/ui/Icon.tsx	`100.00% <0.00%> (ø)`
webapp/javascript/components/RefreshButton.tsx	`100.00% <0.00%> (ø)`
...ponents/FlameGraph/FlameGraphComponent/testData.ts	`100.00% <0.00%> (ø)`
...onents/FlameGraph/FlameGraphComponent/constants.ts	`100.00% <0.00%> (ø)`
...nents/FlameGraph/FlameGraphComponent/Flamegraph.ts	`93.72% <0.00%> (ø)`
... and 16 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ad4722d...0855729. Read the comment docs.

pyroscopebot · 2022-01-08T01:18:08Z

Screenshots

Throughput

Disk Usage

Memory

Upload Errors (Total)

Successful Uploads (Total)

CPU Utilization

Result

	main	pr	diff	threshold
throughput	161.59	160.10	-1.49 (-0.92%)	5%
total items processed	49061.00	49082.00	21.00 (0.04%)	5%

Details

Name	Description	Query for main	Query for pr
throughput		`rate(pyroscope_http_request_duration_seconds_count{handler="/ingest", instance="pyroscope:4040"}[5m])`	`rate(pyroscope_http_request_duration_seconds_count{handler="/ingest", instance="pyroscope-main:4040"}[5m])`
total items processed		`pyroscope_http_request_duration_seconds_count{handler="/ingest", instance="pyroscope:4040"}`	`pyroscope_http_request_duration_seconds_count{handler="/ingest", instance="pyroscope-main:4040"}`

Generated by 🚫 dangerJS against 715eec8

petethepig · 2022-01-08T07:28:43Z

@eh-am maxWorkers one is quite unexpected

…peed-up-local-dev

eh-am · 2022-01-09T14:57:43Z

scripts/webpack/webpack.common.ts

+  .sync(
+    process.env.NODE_ENV === 'production'
+      ? './webapp/templates/*.html'
+      : './webapp/templates/index.html'
+  )


process.env.NODE_ENV isn't really reliable, we could use
https://webpack.js.org/configuration/mode/#mode-none instead

Does this mean in dev mode you won't be able to test other pages? I understand the rationale, but I think this might be confusing when we do want to edit those pages.

It looks like the only thing we do with HtmlWebpackPlugin is this <%= webpack.hash %> Interpolation, and 215ms to do some string replacements is still very very slow imo.

Maybe we could git rid of HtmlWebpackPlugin altogether and instead export webpack.hash to some sort of a file, and then on the go side we would read this file each time we need to render a page and do interpolation via go templates?

I'm not sure how this would work with things like livereload, though.

@eh-am What do you think?

I guess I just remembered we also wanted to get rid of templating on the go side, I guess it would be better to just talk about it on a call.

but I think this might be confusing when we do want to edit those pages.

100% agree, I hope to get rid of this "optimization" soon.

and 215ms to do some string replacements is still very very slow imo.

I read on the webpack docs that stuff like hash isn't very good for production. I didn't profile but I trust them (actually I did but couldn't pin point to the exact problem). I guess we can just not hash for dev? Unless HMR use it for refreshing somehow, but we don't have it so it's a problem for the future.

petethepig

This is awesome! I thought if you want to use esbuild you have to ditch webpack, this is a good middleground.

This also speeds up docker builds (2x improvement for make assets-release step on my machine) which I'm very happy about.

I left a comment about the html pages, but I'm going to merge this one anyway.

* fix type errors * chore: speed up tests as per kulshekhar/ts-jest#259 (comment) using maxWorkers: 1 ends up being faster * set typescript to only build the webapp/javascript dir * use a specific tsconfig just for linting * speed up tests * lint test files as well * use esbuild * optimize local refresh by only building the index page * print to stderr from webpack to make it easier to save stats * inject livereload script to index.html * fix prod build by enabling ts-jest

eh-am added 3 commits January 7, 2022 21:42

fix type errors

5151b72

chore: speed up tests

17ae27a

as per kulshekhar/ts-jest#259 (comment) using maxWorkers: 1 ends up being faster

set typescript to only build the webapp/javascript dir

71cb9c0

eh-am changed the title ~~Chore/speed up local dev~~ chore: speed up local dev Jan 8, 2022

github-actions bot requested review from petethepig and Rperry2174 January 8, 2022 01:06

use a specific tsconfig just for linting

7944643

eh-am marked this pull request as draft January 8, 2022 01:20

eh-am added 3 commits January 8, 2022 13:05

speed up tests

a1558c2

lint test files as well

012f2e3

use esbuild

75226ed

eh-am force-pushed the chore/speed-up-local-dev branch from 62d0385 to 75226ed Compare January 8, 2022 18:40

eh-am added 5 commits January 8, 2022 16:45

optimize local refresh by only building the index page

c94ee08

print to stderr from webpack to make it easier to save stats

8be31af

inject livereload script to index.html

a501e1e

fix prod build by enabling ts-jest

960e2fb

Merge branch 'main' of github.com:pyroscope-io/pyroscope into chore/s…

0855729

…peed-up-local-dev

eh-am marked this pull request as ready for review January 9, 2022 00:55

eh-am commented Jan 9, 2022

View reviewed changes

eh-am mentioned this pull request Jan 9, 2022

feat: export standalone html #691

Merged

petethepig force-pushed the main branch from 5ac4a22 to 65b7372 Compare January 9, 2022 22:14

petethepig approved these changes Jan 9, 2022

View reviewed changes

petethepig merged commit 40c1a21 into main Jan 9, 2022

petethepig deleted the chore/speed-up-local-dev branch January 9, 2022 22:36

eh-am mentioned this pull request Jan 13, 2022

chore: don't calculate webpack hash when in local dev #719

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: speed up local dev #686

chore: speed up local dev #686

eh-am commented Jan 8, 2022 •

edited

github-actions bot commented Jan 8, 2022 •

edited

codecov bot commented Jan 8, 2022 •

edited

pyroscopebot commented Jan 8, 2022 •

edited

petethepig commented Jan 8, 2022

eh-am Jan 9, 2022

petethepig Jan 9, 2022

petethepig Jan 9, 2022 •

edited

eh-am Jan 10, 2022 •

edited

petethepig left a comment

chore: speed up local dev #686

chore: speed up local dev #686

Conversation

eh-am commented Jan 8, 2022 • edited

Local build

Tests

Prod Build

github-actions bot commented Jan 8, 2022 • edited

size-limit report 📦

codecov bot commented Jan 8, 2022 • edited

Codecov Report

pyroscopebot commented Jan 8, 2022 • edited

Screenshots

Result

petethepig commented Jan 8, 2022

eh-am Jan 9, 2022

Choose a reason for hiding this comment

petethepig Jan 9, 2022

Choose a reason for hiding this comment

petethepig Jan 9, 2022 • edited

Choose a reason for hiding this comment

eh-am Jan 10, 2022 • edited

Choose a reason for hiding this comment

petethepig left a comment

Choose a reason for hiding this comment

eh-am commented Jan 8, 2022 •

edited

github-actions bot commented Jan 8, 2022 •

edited

codecov bot commented Jan 8, 2022 •

edited

pyroscopebot commented Jan 8, 2022 •

edited

petethepig Jan 9, 2022 •

edited

eh-am Jan 10, 2022 •

edited