Optimizing incremental bundle time

[mattdesl]

Lately I've been doing some profiling with node-webkit-agent to see where the bundle time is spent on watchify rebundle. It tends to get fairly slow in large projects (current app is more than 1 second per rebundle).

The obvious impact comes from inline source maps. Turning those off reduces the re-bundle time to 100ms.

The next thing I noticed is syntax-error. This runs eval() on the source of every file across your bundle, even if they haven't changed. In my case, re-bundle time goes from ~1000ms (with syntax-error) to ~700ms (without).

Having both of these things only triggered on files that change (rather than across the entire bundle) would probably bring 1000ms re-bundles down to < 100ms.

[zertosh]

webpack has an option (webpack/webpack#168) to create sourcemaps that are eval'ed per module. This saves you having to do the sourcemap stitching that browser-pack does. We could implement something like that directly in module-deps without any changes inside of browserify. opts is already passed into module-deps, so it could even be just adding another option, or setting debug to some other value.

I'm not even sure that syntax-error is needed. The deps step serves as an implicit syntax checker since it parses the source with acorn, which will complain if there's a problem. Between deps and syntax, there is json, unbom, and unshebang. Those are pretty safe. If syntax-error is in fact needed (for some reason that eludes me), we could easily implement a cache there. It doesn't even have to hold on to the entire source, it can just keep hashes of valid code - it's way cheaper to MD5 than to eval.

@jmm @terinjokes If we do need syntax-error, I'd be more than happy to write up a quick cache for it.

[mattdesl]

In my very crude test, I just commented out all the functionality of syntax-error and then re-bundled with watchify. It still seems to catch syntax errors and print them exactly as they were before, so I'm guessing it's all being handled by acorn?

[zertosh]

@mattdesl I can't think of a scenario where module-deps would let broken code through the pipeline. I'm going to run my builds this week with:

b.on('reset', function() { b.pipeline.get('syntax').splice(0, 1); });
b.pipeline.get('syntax').splice(0, 1);

to see how it goes. If it works out, but for whatever reason syntax has to kept around, I'll just keep those two lines in my gulpfile hehe.

I also hacked together a browser-pack that does the eval thing that webpack does. So far so good. It takes almost the same amount of time the first time around. But rebuilds are crazy fast since it only has to rebuild the sourcemaps for the files that changed. Rebuilds on my tester project went from 800ms to 250ms.

This is my branch. To enable the eval'ed modules, instead of passing debug: true in the browserify opts, just pass debug: 'eval'. If anyone is feeling adventurous, they can patch their browser-pack with:

curl "https://raw.githubusercontent.com/zertosh/browser-pack/2997ca7/index.js" > node_modules/browserify/node_modules/browser-pack/index.js

---

How it works:

Modules in regular packs look kinda like this, with one giant sourcemap at the end of the file:

"SoundTitle":[function(require,module,exports){
  /*mycode*/
}, {"react":"react", "truncate":"../lib/truncate"}]

With eval: true, there is no giant sourcemap at the end of the file, and the modules in the pack look like this:

"SoundTitle":[eval("(function(require,module,exports){
  /*mycode*/
  //# sourceMappingURL=data:application/json;base64,eyJ2Z.....
})"), {"react":"react", "truncate":"../lib/truncate"}]

Of course this is terrible for production, but for fast dev'ing, it works. Any tools that depend on being able to read the module source after the build will break - like bundle-collapser. But I doubt anyone runs bundle-collapser while dev'ing.

One thing I don't like, is that the dev tools fill up with data url requests. But you can hide those, so eh.

[ghost]

I think syntax-error might've been necessary in the past because of either vague messages from esprima without line numbers or some esprima-fb related jsx thing. This might not be a problem now with acorn.

[mattdesl]

Woah @zertosh amazing work. 👏 I just tested your branch and also removed syntax-error from the pipeline, and my 800ms re-bundle time is down to 140ms.

[terinjokes]

@zertosh do you find that the browser parses the source maps at about the same speed?

[zertosh]

@substack @mattdesl Turns out that there are 3 cases where you need syntax-error:

1. Dep graph leaf nodes. module-deps->detective skips the acorn parse as part of an optimization when via a simple string match it can't find a require.
2. Deps with noparse - Though this one I'm kinda iffy about. Maybe it's a feature not to syntax check these modules?
3. json. While looking at this one, I noticed that we were missing Sanitize json modules #1211

[zertosh]

@terinjokes Ah. Interesting. I don't even know how I would measure that. Any ideas?

[terinjokes]

@zertosh I do not. I've asked if there's an internal mechanism.

I already have a long parse time for one project, and I'm interested if this change would make it larger.

[zertosh]

hey @mattdesl, I figured out how to make the sourcemaps super fast w/o having to do any eval'ing - you still get inline sourcemaps. try it out and let me know what you think: (cd node_modules/browserify && npm i substack/browser-pack#sm-fast). My rebuild times were cut almost in half.

https://github.com/substack/browser-pack/tree/sm-fast

[mattdesl]

@zertosh that looks awesome.. will do some testing

[PHaroZ]

@zertosh +1 for your solution, thanks a lot !

[mattdesl]

@zertosh Any chance of that being merged into browser-pack? How can we help move it along? Just more testing?

[joshrtay]

+1 Hope to see this merged soon.

[ashaffer]

@mattdesl @zertosh Is there a way you guys are testing this branch? I'd like to test it and also use it locally for development. Should I just fork browserify or is there some trick to getting browserify to use this?

I mean, other than just installing it in node_modules where it'll get blown away.

[ashaffer]

Any updates on this?

[danielcardoso5]

I'm using the branch provided by @zertosh (https://github.com/substack/browser-pack/tree/sm-fast) in my build. I'm using Yarn resolutions to force browserify to use that branch as a dependency. My incremental bundles still take some time but still an improvement with that branch. A lot of the time is still due to sourcemaps, without sourcemaps it's really fast.

Is there a better way to do it at the moment? Does switching to webpack give any improvements on incremental build times when using sourcemaps?