HTTP2 support

[wmertens]

Idle musing prompted by http://pca.st/p47h :

What would be the best webpack configuration for fast HTTP2 loading? Simply put each processed file in a separate chunk? Maybe provide loading hints for images?

It would be nice to have two builds, serving the HTTP2 build when appropriate…

[altano]

It's not good enough to stick the files in separate chunks. You have to have a web server that is capable of server-pushing or server-hinting the dependent assets when something is requested from the web server, and you have to have a way of telling it what to send.

If someone wrote a webpack plug-in to put every asset in its own chunk, AND generate a manifest of all dependent assets, AND modified a webserver to be able to read that manifest and do server-push for dependencies, then you'd be all set for HTTP2.

None of this work is done yet as far as I can tell. Someone should go do it :)

[wmertens]

Is it really necessary to do server push? Isn't that an optimization,
sometimes resulting in too much data being sent?

Doesn't HTTP2 make multiple requests by the browser almost equivalent to
concatenation?

On Tue, Jul 7, 2015, 18:52 Alan notifications@github.com wrote:

It's not good enough to stick the files in separate chunks. You have to
have a web server that is capable of server-pushing or server-hinting the
dependent assets when something is requested from the web server, and you
have to have a way of telling it what to send.

If someone wrote a webpack plug-in to put every asset in its own chunk,
AND generate a manifest of all dependent assets, AND modified a webserver
to be able to read that manifest and do server-push for dependencies, then
you'd be all set for HTTP2.

None of this work is done yet as far as I can tell. Someone should go do
it :)

—
Reply to this email directly or view it on GitHub
#1223 (comment).

Wout.
(typed on mobile, excuse terseness)

[altano]

Isn't that an optimization, sometimes resulting in too much data being sent?

It can lead to that, but if you do what I outlined above, you would never send too much data. If script A is requested, and X Y and Z are required() dependencies according to webpack, you need to send all 4 scripts to be able to execute A.

Doesn't HTTP2 make multiple requests by the browser almost equivalent to concatenation?

Not at all. Imagine you request script A, which requires script B, which requires script C. The browser doesn't know about script B until it full downloads and executes script A. Same for C. There's a round-trip between each download, as a result.

HTTP2 will make it so that if script A requires script B, C, and D, the browser can request all 3 and receive all 3 with only one round-trip. In HTTP1 that would be 4 round-trips no matter what, with HTTP2 it would just be two round-trips, but that isn't nearly as fast as the server preemptively pushing down all the resources with just one round-trip.

[wmertens]

sure, but in the case of already-cached files the browser has to send a
reset and depending on latency a good chunk of unnecessary data might have
been sent before the server stops sending…

On Sat, Jul 11, 2015, 22:51 Alan notifications@github.com wrote:

Isn't that an optimization, sometimes resulting in too much data being
sent?

It can lead to that, but if you do what I outlined above, you would never
send too much data. If script A is requested, and X Y and Z are required()
dependencies according to webpack, you need to send all 4 scripts to be
able to execute A.

Doesn't HTTP2 make multiple requests by the browser almost equivalent to
concatenation?

Not at all. Imagine you request script A, which requires script B, which
requires script C. The browser doesn't know about script B until it full
downloads and executes script A. Same for C. There's a round-trip between
each download, as a result.

HTTP2 will make it so that if script A requires script B, C, and D, the
browser can request all 3 and receive all 3 with only one round-trip, but
that isn't nearly as fast as the server preemptively pushing down all the
resources.

—
Reply to this email directly or view it on GitHub
#1223 (comment).

Wout.
(typed on mobile, excuse terseness)

[cesarandreu]

One idea is to have the entry point script (or maybe an inline-script in the index.html?) include a list of all the files in the depended-on bundles. Wouldn't that tackle part of the problem in a reasonably simple way?

[thanzen]

+1

[capaj]

or you could just use JSPM/SystemJS, which has depcache support from the day 1. Only package manager truly ready for HTTP2 world.

[polarathene]

@capaj From what I can gather, that only sends multiple requests for modules to the server. The initial HTML/JS still have to be downloaded/parsed before it can begin that. Also depCache is being deprecated: depCache is being deprecated for a straightforward call of Object.keys(trace.tree).. While sending multiple requests to the server for the modules takes advantage of HTTP/2 multi-plexing, it does not benefit from server-push or server-hints which will send the resources to the client with the initial HTML request/response which reduce round trips.

[capaj]

@polarathene depCache method is an implementation detail. The feature I was talking about is depcache generated by JSPM, when invoked from commandline jspm depcache ... as documented here: https://github.com/jspm/jspm-cli/blob/master/docs/production-workflows.md#creating-a-dependency-cache

It is NOT deprecated and I suspect it won't ever be.
It works by traversing whole tree of dependencies and then storing the map in config.js. This file then can be used on the server/client to get/push all of the dependencies up front.

[polarathene]

@capaj See the github issue I linked to, the blue text specifically highlights the last comment by the owner of SystemJS. It's fairly recent so I'm pretty sure the intention is to deprecate depCache.

Your link mentions that once the initial js file is loaded/parsed by the client(1 request for html, 1 request for the initial js file once html is parsed) then it will request from the server for all dependencies(many requests which is fine on HTTP/2 or SPDY supported browsers provided the server supports those protocols). There is nothing special going on there, using the dependency graph to fetch all the dependencies in advance from then is neat and takes advantage of multi-plexing to download the assets faster, however server-push and server-hints are server based.

With the server support you'd link your js assets with the html request(1 round trip), or individually hint/push assets with their dependencies as they're requested(eg send html with linked assets above the fold, but request the rest as normal via lazy loading). This requires the server to support push/hints for the protocol along with knowing what assets to link with requests for push/hinting in the headers sent to the client.

[capaj]

@polarathene yes the method is depracated, but it does not mean that depcache is depracated in JSPM. I know @guybedford better than you, you don't have to tell me who he is. He is awesome.

Doing a server push library which will use JSPM's config.js to push dependencies to client is trivial- if I actually had HTTP2 enabled website, I would make it myself in less than one evening. Unfortunately I don't.

[AlistairB]

I want webpack to allow turning off concatenation and being able to spit out all the files that have been referenced by a single page as simple script referenes. This is what makes sense to me in a HTTP 2.0 world.

[polarathene]

Alistair, webpack can split the bundle into multiple files. You still only reference one in a script tag though, when those scripts are needed they're requested, this makes no difference with http 2. If you want to use features beyond making many quests you need to do some server sided hinting or push.

[nicoqh]

@AlistairB how about splitting the bundle into multiple files/chunks and using the ManifestPlugin? You could reference the generated manifest file server-side and create the required script tags.

[kevinSuttle]

Any update on this?

[sokra]

It's on the roadmap... The plan is:

• automatically split the bundle into many chunks (i. e. by module age)
• emit a push manifest for module assets

[timatooth]

I'm especially interested in how the the push manifest might be structured and how that could be integrated into CDNs such as MaxCDN, Akamai, CloudFlare etc. Most do not support preemptive server pushing of assets to clients thus far.

Or running your own static server: are there options for nginx or mod_http2 to make use of manifests to offload the responsibility of an app developer needing to know which assets need pushed down to the client?

[AlistairB]

I think the ideal scenario is individual files included by script tags with a hash of the contents in the filename / path which can be server pushed via a manifest. At the same time CDNed resources are just included from the CDN.

@sokra Are they arbitrary chunks? That wouldn't be ideal as with each build all of the chunks would be invalidated, unlike using the actual files with a hash.

[kevinSuttle]

FWIW: https://github.com/GoogleChrome/simplehttp2server https://www.youtube.com/watch?v=qx9tHwhjkHs

[aldendaniels]

+1 to @AlistairB's comment. Many CDNs and modern browsers already support HTTP2, so I for one would use this functionality immediately.

I think the only initial requirements are:

1. Generate one versioned output file (chunk) per input file
   @sokra Any chance this is already supported?
2. Generate corresponding script tags which can be injected into HTML
   (supported using html-webpack-plugin)

---

Additional long-term improvements would include:

1. HTTP2 push support
2. Staged loading of assets... like chunking, but at the protocol level. This way you could load all assets needed for the initial page in one go (via script tags) and then load additional asset collections via an async loading API.

[kevinSuttle]

I would hope that this be a top priority @sokra. Is there a roadmap?

[sokra]

Yes. It''s the next feature on the list. 80% done...

[sokra]

Ok I've got an experimental version of it:

https://github.com/webpack/webpack/tree/master/examples/http2-aggressive-splitting

You need to do the following:

• add new webpack.optimize.AggressiveSplittingPlugin()
• use [chunkhash] in the filenames
• add multiple script tags to the HTML page (see stats.entrypoints["main"] for a list)
• use records to store chunk/module ids and the splittings
• This is production only, don't use it in development

It then generates a HTTP2 optimized version.

Advantages:

• It allows to cache parts for the application (without 304)
• It loads parts for the application in parallel
• Still allow to uglify and gzip effectively

Disadvantages:

• It causes more requests for non HTTP2 clients
• You need infrastructure to keep the records
• It could make builds slower

[sokra]

@ampedandwired webpack now emits information about entrypoints into the stats. I. e. which assets are required for an entrypoint and in which order.

entrypoints: {
  main: {
    chunks: [1, 2, 4],
    assets: ["123.js", "456.js", "789.js"]
  }
}

[kevinSuttle]

@sokra Can you elaborate on this?

You need infrastructure to keep the records

[sokra]

webpack need the keep "state" between compilations. It's called the records, which are saved and loaded from a file (specified by recordsPath). You need to be able to keep this file between compilation, which may be more difficult if you using a build server.

[kevinSuttle]

Ah ok. Thanks!

[MagicDuck]

@sokra I am in the process of webpackifying a large app with around 4000 modules.
A big issue I am running into is the incremental build speeds which are critical to be low in order to have a pleasant development process. I've applied all the optimizations I could find online, but could only get it to 8 - 19s. It seems to have a variance depending on the size of the chunk the module I am editing belongs to. My app has a lot of legacy with deps that are hard to break up, we're gradually working on it though.
In any case, what I was wondering is if http2 could help with the incremental build speeds since the chunks are always small. Would it not keep the speed pretty much constant no matter how your app grows? That is also assuming that webpack-dev-server gets http2 support.
Btw, is there a version of webpack in npm that has the experimental support you mention above?

[imvetri]

Question 1 : Can someone judge whether my understanding is right or wrong?

https web: will help multiple files to be transfer between server and client in a single connection.

http web multiple files will be stitched together and sent in a single connection.

My theory is that bundling wont be needed in case of http2.

Question 2: Is it fine to get these kinds of doubts to be clarified here in github issue thread? I feel i may have deviated the thread.

[rarkins]

Right now the webpack default approach is "bundling" and exceptions to that use terms like "splitting". To me this sounds philosophically reverse of a future HTTP2 approach where the default should be not bundling. In most cases I don't want to "split my bundles" - I want no bundling as default and to optionally bundle. Can webpack support this reverse preference/philosophy?

[pwang2]

@rarkins brought a very good point here.

For static assets, when it comes to HTTP2, we like caching and multiplexing; when it comes to bundler, we like the composition to hide the detail. I want to lay some groundwork by sharing our practice first.

In our practice, we like both. We use webpack as an emitter to get those packed files. We use HTTP2 as a transporter to get packed file to user browser. Not only this, we also want it lives with a package manager doom(we choice bower at no bias as it does the job). The emitted packed files from webpack only contain the implementation code and the stuffs not shareable. Reference to the dependencies are preserved with webpack externals and declared in the bower.json.

When comes to the integration application, as an organization, we want all shareable package be shared in all products via a CDN. We have a publish service to publish all bower component to CDN server. All publish files are versioned and cached by maxAge:bignumber. In browser, we use a loader to download the main files by package name and add the bootstrap/configuration code as a callback. With the help of HTTP2 and CDN, the transporter did a great job.
As a plan B, the loader is pluggable, application developer could live with the old grunt/gulp pack workflow as well, only to generate a different index.html which contains the packed bundles(vendor.js, helper.js, app.js).

This has been validated as a pretty smooth workflow in new app building. The shortcoming is, it can not be easily added to existing aged application. In enterprise, it is hard to promote as you have to make all peer teams agree. Most of it become to sociological problems.

Back to webpack here, I think we could control webpack-dev-server suspend before bundling work and emit the packed sub application chunk to memfs only. Reference to the dependencies are preserved with webpack externals, and be served from node_modules/bower_components. Then in prod mode, memfs and the static files will be replaced with a CDN publish service.

It is more enterprise workflow specific and not as general as webpack's work here. But this solves some problem brought by bundle and the practice could be extended to production practice as well in a HTTP2 user case.

ps: I love @sokra 's work a lot and I am a huge webpack fan.

[webpack-bot]

This issue had no activity for at least half a year.

It's subject to automatic issue closing if there is no activity in the next 15 days.

[webpack-bot]

Issue was closed because of inactivity.

If you think this is still a valid issue, please file a new issue with additional information.

[sharmad-nachnolkar]

@sokra I am trying to use this plugin with webpack. It is able to split bundles into smaller chunks but when I am loading these script files in html, these modules are not being executed. Any idea?

[njgraf512]

Does anyone know of companies that are using aggressive splitting and HTTP/2 to serve their web content?

[Lynn-cc]

Ok I've got an experimental version of it:

https://github.com/webpack/webpack/tree/master/examples/http2-aggressive-splitting

You need to do the following:

• add new webpack.optimize.AggressiveSplittingPlugin()
• use [chunkhash] in the filenames
• add multiple script tags to the HTML page (see stats.entrypoints["main"] for a list)
• use records to store chunk/module ids and the splittings
• This is production only, don't use it in development

It then generates a HTTP2 optimized version.

Advantages:

• It allows to cache parts for the application (without 304)
• It loads parts for the application in parallel
• Still allow to uglify and gzip effectively

Disadvantages:

• It causes more requests for non HTTP2 clients
• You need infrastructure to keep the records
• It could make builds slower

The plugin splits chunks by setting min and max size. But There's a stream number limit parameter in HTTP2: SETTINGS_MAX_CONCURRENT_STREAMS. And my server sets it to 128. Then how could I maximize the initial/async chunk number near to the max concurrent streams ?