cacheDirectory babel-loader #5436

MichaelrMentele · 2022-05-05T01:58:43Z

Noticed when investigating #5231 that we aren't caching transpilation when using the babel-loader plugin that glues together webpack and babel.

The impact of this is that we are potentially retranspiling ALL of our files when often some tiny fraction of them actually need to be transpiled (when we call bebel with webpack through babel-loader)

Seems to have a net O(10%) speed up for net build time for the redwood web side. Course babel-loader itself is only one part of that process -- according to the docs this should give an O(2x) speed boost when cached so working backwards sounds like our overall transpilation time is about ~6 s or roughly 20% of overall build time -- for a relatively small redwood project on my machine.

Redwood Build Time Before:

➜  web git:(main) ✗ python -m timeit "__import__('os').system('yarn workspace web rw build web')
"
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
1 loop, best of 5: 42.6 sec per loop

Redwood Build Time After:

➜  web git:(main) ✗ python -m timeit "__import__('os').system('yarn workspace web rw build web')
"
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
  ✔ Cleaning Web...
  ✔ Cleaning Web...
  ✔ Building Web...
  ✔ Prerendering Web...
1 loop, best of 5: 39.2 sec per loop

netlify · 2022-05-05T01:58:48Z

✅ Deploy Preview for redwoodjs-docs canceled.

Name	Link
🔨 Latest commit	`aa90c49`
🔍 Latest deploy log	https://app.netlify.com/sites/redwoodjs-docs/deploys/62742ed710cbd100095af8f5

…com/LeftLaneSoftware/redwood into rw-build-times-babel-loader-caching

MichaelrMentele · 2022-05-05T17:27:20Z

Some reference plugins I used for profiling -- note the built in profiler does NOT work -- throws when dropped into chrome debugger. @dac09 mentioned it was neever updated to work with webpack 5.

const webpack = require('webpack')
const WebpackBar = require('webpackbar')
const path = require('node:path')
const SpeedMeasurePlugin = require('speed-measure-webpack-plugin')

/** @returns {import('webpack').Configuration} Webpack Configuration */
module.exports = (config, { mode }) => {
  if (mode === 'development') {
    // Add dev plugin
  }

  // Add custom rules for your project
  // config.module.rules.push(YOUR_RULE)
  // config.plugins.push(new webpack.debug.ProfilingPlugin())
  // config.plugins.push(
  //   new webpack.debug.ProfilingPlugin({
  //     outputPath: path.join(__dirname, 'profiling/profileEvents.json'),
  //   })
  // )

  const smp = new SpeedMeasurePlugin({
    granularLoaderData: true,
    outputFormat: 'human',
    // outputFormat: 'humanVerbose',
    loaderTopFiles: 20,
  })
  config.plugins.push(smp)

  // Add custom plugins for your project
  // config.plugins.push(YOUR_PLUGIN)
  // config.devtool = false
  // config.cache = true

  config.devtool = 'source-map'
  config.cache = true
  // config.watch = true
  // config.plugins.push(new WebpackBar({ profile: true, color: 'red' }))

  // disable source map saves about 1-2 seconds
  // enabling caching saves

  // config.watch = true

  return config
}

MichaelrMentele · 2022-05-05T17:31:14Z

Running SMB -- where do the other 10 s go?

➜  leftlane_app git:(main) ✗ yarn rw build web --verbose
[12:29:56] Cleaning Web... [started]
[12:29:56] Cleaning Web... [completed]
[12:29:56] Building Web... [started]
(node:50101) [DEP_WEBPACK_COMPILATION_NORMAL_MODULE_LOADER_HOOK] DeprecationWarning: Compilation.hooks.normalModuleLoader was moved to NormalModule.getCompilationHooks(compilation).loader
(Use `node --trace-deprecation ...` to show where the warning was created)


 SMP  ⏱
General output time took 25.81 secs

 SMP  ⏱  Loaders
modules with no loaders took 8.02 secs
  module count                                                                                                     = 12542
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@mui/material/Popover/Popover.js                    = 0.204 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@mui/material/Paper/paperClasses.js                 = 0.201 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@mui/icons-material/esm/index.js                    = 0.2 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_unicodeToArray.js                           = 0.193 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseToString.js                             = 0.193 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_asciiToArray.js                             = 0.189 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseSlice.js                                = 0.188 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_defineProperty.js                           = 0.186 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseFor.js                                  = 0.186 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/noop.js                                      = 0.185 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_trimmedEndIndex.js                          = 0.185 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/core-js-pure/modules/es.date.now.js                 = 0.184 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_Set.js                                      = 0.184 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@popperjs/core/lib/dom-utils/getWindowScrollBarX.js = 0.183 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_setCacheHas.js                              = 0.183 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseIndexOf.js                              = 0.183 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_MapCache.js                                 = 0.182 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_setCacheAdd.js                              = 0.182 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_baseProperty.js                             = 0.181 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/lodash/_basePropertyDeep.js                         = 0.181 secs
babel-loader took 1.71 secs
  module count                                                                                                                   = 70
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Stage/NewStage/NewStage.tsx                                 = 0.263 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/VehicleCard/VehicleCard.tsx                                 = 0.256 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/StageCard/StageCard.tsx                                     = 0.255 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/NewVehicleForm/VinLookupCell/VinLookupCell.tsx              = 0.254 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/NewVehicleForm/VinForm/VinForm.tsx                          = 0.253 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/EditPipelineCell/EditPipelineCell.tsx              = 0.252 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/PipelineCell/PipelineCell.tsx                      = 0.251 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/NewPipeline/NewPipeline.tsx                        = 0.25 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/Pipeline/PipelinesCell/PipelinesCell.tsx                    = 0.249 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/App.tsx                                                                = 0.248 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/api/src/lib/logger.ts                                                          = 0.17 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/SettingsPage/SettingsPage.tsx                                    = 0.154 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/UploadVehicleImageComponent/UploadVehicleImageComponent.tsx = 0.153 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/SignupPage/SignupPage.tsx                                        = 0.138 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/VehicleDetailPage/VehicleDetailPage.tsx                          = 0.135 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/components/RemoteImageCell/RemoteImageCell.tsx                         = 0.135 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/VehiclePipelinePage/VehiclePipelinePage.tsx                      = 0.132 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/layouts/DefaultLayout/DefaultLayout.tsx                                = 0.125 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/Pipeline/EditPipelinePage/EditPipelinePage.tsx                   = 0.124 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/pages/Pipeline/PipelinePage/PipelinePage.tsx                           = 0.121 secs
mini-css-extract-plugin, and
css-loader took 0.214 secs
  module count                                                         = 2
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/scaffold.css = 0.212 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.css    = 0.203 secs
css-loader took 0.058 secs
  module count                                                                                                                                                                                                                                                                                                               = 2
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.css.webpack[javascript/auto]!=!/Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!/Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.css       = 0.057 secs
  /Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/scaffold.css.webpack[javascript/auto]!=!/Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!/Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/scaffold.css = 0.016 secs
null-loader took 0.025 secs
  module count                                                                                       = 1
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/@redwoodjs/router/dist/splash-page.js = 0.025 secs
html-webpack-plugin took 0.01 secs
  module count                                                                                                                                                      = 1
  /Users/michaelmentele/Dev/leftlane/leftlane_app/node_modules/html-webpack-plugin/lib/loader.js!/Users/michaelmentele/Dev/leftlane/leftlane_app/web/src/index.html = 0.01 secs



assets by path static/js/*.js 1.76 MiB
  assets by chunk 490 KiB (id hint: vendors)
    asset static/js/709.e1832718.chunk.js 378 KiB [emitted] [immutable] [minimized] [big] (id hint: vendors) 2 related assets
    asset static/js/846.3512a8f1.chunk.js 57.6 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
    asset static/js/857.da72520d.chunk.js 32.4 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
    asset static/js/961.c36d2134.chunk.js 16.1 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
    asset static/js/248.65d25381.chunk.js 6.54 KiB [emitted] [immutable] [minimized] (id hint: vendors) 2 related assets
  + 27 assets
asset static/css/app.1fe0c5b7.css 5.83 KiB [emitted] [immutable] [minimized] (name: app)
asset build-manifest.json 4.71 KiB [emitted]
asset mockServiceWorker.js 4.18 KiB [emitted] [from: public/mockServiceWorker.js] [copied] [minimized]
asset README.md 1.9 KiB [emitted] [from: public/README.md] [copied]
asset favicon.png 1.7 KiB [emitted] [from: public/favicon.png] [copied]
asset index.html 936 bytes [emitted]
asset robots.txt 24 bytes [emitted] [from: public/robots.txt] [copied]
Entrypoint app [big] 659 KiB (2.53 MiB) = static/js/runtime-app.62d0b37a.js 4.44 KiB static/css/app.1fe0c5b7.css 5.83 KiB static/js/app.60e81984.js 649 KiB 2 auxiliary assets
orphan modules 8.94 MiB (javascript) 2.4 KiB (runtime) [orphan] 11589 modules
runtime modules 13.2 KiB 21 modules
cacheable modules 4.4 MiB (javascript) 6.37 KiB (css/mini-extract)
  modules by path ../node_modules/ 3.21 MiB 1001 modules
  modules by path ./src/ 1.19 MiB (javascript) 6.37 KiB (css/mini-extract)
    modules by path ./src/pages/ 1.1 MiB 24 modules
    modules by path ./src/components/ 21.4 KiB
      ./src/components/UI/LoadingSpinner/LoadingSpinner.tsx 253 bytes [built] [code generated]
      ./src/components/VehicleSummaryCell/VehicleSummaryCell.tsx 9.6 KiB [built] [code generated]
      ./src/components/RemoteImageCell/RemoteImageCell.tsx 2.72 KiB [built] [code generated]
      + 3 modules
    modules by path ./src/*.css 6.37 KiB
      css ../node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!./src/scaffold.css 6.37 KiB [built] [code generated]
      css ../node_modules/@redwoodjs/core/node_modules/css-loader/dist/cjs.js??ruleSet[1].rules[0].oneOf[3].use[1]!./src/index.css 0 bytes [built] [code generated]
    ./src/App.tsx + 23 modules 72.4 KiB [built] [code generated]

WARNING in asset size limit: The following asset(s) exceed the recommended size limit (244 KiB).
This can impact web performance.
Assets:
  static/js/app.60e81984.js (649 KiB)
  static/js/180.163d65f8.chunk.js (372 KiB)
  static/js/709.e1832718.chunk.js (378 KiB)

WARNING in entrypoint size limit: The following entrypoint(s) combined asset size exceeds the recommended limit (244 KiB). This can impact web performance.
Entrypoints:
  app (659 KiB)
      static/js/runtime-app.62d0b37a.js
      static/css/app.1fe0c5b7.css
      static/js/app.60e81984.js


webpack 5.70.0 compiled with 2 warnings in 25814 ms
Creating 200.html...
[12:30:27] Building Web... [completed]
[12:30:27] Prerendering Web... [started]
[12:30:27] Prerendering Web... [completed]
➜  leftlane_app git:(main) ✗

dac09 · 2022-05-10T18:21:51Z

Hey @MichaelrMentele - appreciate all the effort into these PRs - but something I wanted to point out - the jest tests don't actually use webpack - and therefore doesn't use babel loader. Yes rw build performance may improve with this cache directory setting, but you can see that its quite minimal!

I would say the two files that are most likely the culprits for the jest initialisation time are these:

In particular, I have a feeling it may be the loading of the project mocks in the beforeEach in jest setup.

jtoar · 2022-07-07T06:20:26Z

@dac09 Michael mentioned in the OP that this was for build, not for test. And it seems that it does improve build time, even if it’s a little. We’ve certainly merged smaller performance gains before. So thoughts on getting this one in? It's not critical, but it'd nice to bring this one to a close one way or another.

jtoar · 2022-07-28T08:52:48Z

@MichaelrMentele I discussed with @dac09 the other day. We're going to close this one cause we're not sure that the possible performance gain is worth the change webpack config, which just comes with some unknowns for us. If we had more tests around our webpack config, we'd be more confident in this. Sorry that this was open for so long—thanks for taking the time to do the research you did and contribute!

cacheDirectory babel-loader

257fb37

MichaelrMentele marked this pull request as ready for review May 5, 2022 02:24

MichaelrMentele added 3 commits May 4, 2022 21:24

Merge branch 'main' into rw-build-times-babel-loader-caching

e75ea4c

make it overridable

7dd80ba

Merge branch 'rw-build-times-babel-loader-caching' of https://github.…

ea67b8c

…com/LeftLaneSoftware/redwood into rw-build-times-babel-loader-caching

Merge branch 'main' into rw-build-times-babel-loader-caching

c78b43a

Merge branch 'main' into rw-build-times-babel-loader-caching

aa90c49

dac09 added the release:chore This PR is a chore (means nothing for users) label May 10, 2022

jtoar linked an issue May 20, 2022 that may be closed by this pull request

Jest Web Test Run Initialization is Slow O(10x) Slower than it Should Be #5231

Closed

jtoar assigned dac09 and jtoar Jul 7, 2022

jtoar closed this Jul 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cacheDirectory babel-loader #5436

cacheDirectory babel-loader #5436

MichaelrMentele commented May 5, 2022 •

edited

netlify bot commented May 5, 2022 •

edited

MichaelrMentele commented May 5, 2022

MichaelrMentele commented May 5, 2022

dac09 commented May 10, 2022

jtoar commented Jul 7, 2022

jtoar commented Jul 28, 2022

cacheDirectory babel-loader #5436

cacheDirectory babel-loader #5436

Conversation

MichaelrMentele commented May 5, 2022 • edited

netlify bot commented May 5, 2022 • edited

✅ Deploy Preview for redwoodjs-docs canceled.

MichaelrMentele commented May 5, 2022

MichaelrMentele commented May 5, 2022

dac09 commented May 10, 2022

jtoar commented Jul 7, 2022

jtoar commented Jul 28, 2022

MichaelrMentele commented May 5, 2022 •

edited

netlify bot commented May 5, 2022 •

edited