Resolvers and transformers

[Rich-Harris]

Rollup needs to be capable of resolving external modules. The obvious scenario is the one in which external modules live in a node_modules folder, and (at least in some cases) have a jsnext:main field pointing to an ES6 module. Also, it would be good to support jspm_packages.

But we might not want to bundle external modules, so it shouldn't necessarily be the default. Maybe it looks like this:

rollup -i main.js -o bundle.js --resolve npm jspm

rollup.rollup( 'main.js', {
  resolvePath: [ rollup.npm, rollup.jspm ]
}).then( function ( bundle ) {
  /* ... */
});

The other factors to consider are a) whether to try and convert legacy formats to ES6 and b) whether to support loader plugins (e.g. import 'styles.css!' or import data from 'data.json!' or whatever).

Need to bear in mind how all this interacts with transform steps, caches and so on.

[curran]

+1 it would be awesome to be able to use JSPM for package management and Rollup for bundling. Thank you for this amazing work.

[Rich-Harris]

@curran Thanks for giving it a go! Do you happen to have a JSPM-based project online anywhere? I'm not really that familiar with it so it would be useful to see it in the wild so I can get my head round where it puts everything.

One potential wrinkle: IIRC SystemJS handles all different module types behind the scenes, whereas Rollup can only deal with ES6. So it probably wouldn't be a drop-in replacement.

[curran]

Indeed! I am excited to see the granular strategy of Rollup, it seems so right.

I'm still not sure if I will adopt JSPM, and I am also new to it, but I created an example project to test out the tooling - jspm-mocha-example. The gh-pages branch there has a project that depends on the master branch as a JSPM package. This project uses AMD modules though, not ES6.

Also, I have these two experimental projects that might be more interesting for you because they are using ES6 modules. I want one to depend on the other, and I want to expose both as standalone UMD bundles: reactive-model and graph. Reactive-model is set up to depend on graph via JSPM. With this setup, running jspm install from within reactive-model populates the jspm_packages as follows:

$ cd reactive-model
$ tree jspm_packages/
jspm_packages/
├── es6-module-loader.js
├── es6-module-loader.js.map
├── es6-module-loader.src.js
├── github
│   └── curran
│       ├── graph@0.0.3
│       │   └── graph.js
│       └── graph@0.0.3.js
├── system.js
├── system.js.map
└── system.src.js

Also, JSPM/SystemJS uses a mapping file config.js that is used for resolving module paths.

$ cat config.js 
System.config({
  "baseURL": "/",
  "paths": {
    "*": "*.js",
    "github:*": "jspm_packages/github/*.js"
  }
});

System.config({
  "map": {
    "graph": "github:curran/graph@0.0.3"
  }
});

Within Graph, the jspm property inside package.json tells JSPM/SystemJS where to look for the main module file.

$ cd graph
$ cat package.json 
{
  ...normal package.json stuff omitted...
  "jspm": {
    "directories": {
      "lib": "src"
    },
    "main": "graph",
    "format": "es6"
  }
}

It's such a struggle to figure out the right set of tools to use. At this point I'm just trying out everything.

Hope this info is helpful to you. All the best with Rollup.

[guilhermeaiolfi]

I'm using JSPM and it's definitely the way forward. The bundling part of JSPM is the least great part. So I think rollup would fit perfectly.

There is this great article that describe how to setup JSPM and how to use it: http://developer.telerik.com/featured/choose-es6-modules-today/

The final code is in: https://github.com/codylindley/jspm-startup. That's the best way to start with JSPM.

I hope to use rollup+JSPM in the future.

[Rich-Harris]

@curran

At this point I'm just trying out everything

Thanks for sharing those projects. I know exactly how you feel! We're definitely in the Wild West phase of this stuff - it feels like every project requires a different set of hacks.

@guilhermeaiolfi Good to see you here! Glad to hear you think highly of JSPM as a package manager - I'd definitely like for Rollup to be able to bundle JSPM-based projects. The fact that the jspm-startup project's config.js is 169 lines long makes me very nervous, and is basically one of the main reasons I still use and recommend npm (it Just Works™), but I've only heard good things from people who've embraced JSPM and got over the initial hump.

In fact, I'm going to cc @guybedford (JSPM creator) in case he has any thoughts on this stuff. (Hi Guy, sorry for spamming you! Thought this was a thread that might be of interest to you - Rollup is an ES6 module bundler that aims to produce maximally efficient bundles, we'd like it to work out of the box with System.config etc. Would love to compare notes!)

[guybedford]

@Rich-Harris thanks for copying me in here! Here are some of my opinions on these matters:

• In the readme you say:

Using a folder structure to define an interface is a bad idea

I completely disagree on this - just like objects in JS have a private interface that users can abuse, the same applies to modular library file systems. Just because private things can be abused does not mean that a public file interface is not an important use case. It is because of the lack of easy modularity systems today that we think of libraries as "one main", but in ES6 we need to open up to the concept of many public entry points, as it is the only way to allow bundle sizes to be reduced in future, just loading the parts you need. Babel's use of the core-js runtime in a modular way is the primary example of this today (and perhaps as well lodash).

• In terms of an external resolver as discussed in this issue - the most general resolver hook is a function:

rollup.rollup( 'main.js', {
  resolvePath: function(relativePath, parentPath) { return path;  }
}).then( function ( bundle ) {
  /* ... */
});

Such a function would then be able to plugin into a jspm resolver for example allowing users to make folded bundles through this library which would be amazing for our users.

The tree shaking of this library is a critically important feature for the success of ES6. In theory the approach along the lines of this project can always provide better file size and performance over jspm, but the point is to see them as two sides of the same coin as opposed to competing solutions. I would love to be able to offer this option out of the box in jspm like a jspm static-bundle command - if a resolver can work to do this that would be interesting to see. Out of interest - have you looked at the optimizations made in the bundling of ES6-Module-Transpiler? There was some great work there, it is a shame to see it stagnating. I really look forward to seeing where you take this project.

[Rich-Harris]

@guybedford thanks for weighing in!

There are currently two hooks for resolving - resolvePath, which you can override but probably don't want to, and resolveExternal which you definitely might want to override. resolvePath defers to resolveExternal if the path doesn't begin with ., and if the module ID wasn't included in options.external (which says 'don't bother bundling these modules').

At the moment resolveExternal defaults to this function which only knows about node_modules and jsnext:main. So in theory it's just a case of writing a function that knows about jspm_modules and System.config, notwithstanding the small matter of converting AMD/CJS to ES6. Would like to take a crack at this soon - if you could point me in the right direction that'd be great. (Also, any advice about converting AMD/CJS would be warmly received!)

in ES6 we need to open up to the concept of many public entry points, as it is the only way to allow bundle sizes to be reduced in future

I strongly agree, though my understanding was that these entry points were to be defined by named exports rather than the folder structure. I take your point about abusing private interfaces, but I actually think there are more subtle and pernicious disadvantages to using a folder structure as an interface, such as the way it forces you to organise your code. To take D3 as an example, you have functions like d3.ascending currently defined in src/arrays/ascending, because that's more logical from a code organisation point of view, and if the folder structure defined the API it would be impossible to reorganise modules without a major version bump. (As it happens, the next version of D3 will have a custom build process via d3-bundler which wraps Rollup.)

It's an important conversation to have though, since I think there needs to be a common approach to this question - my understanding of the issue is based on a comment from @caridy:

One note about jsnext:main that I forgot to mention before :), it does not support specifying paths within a module, saying, if the value of jsnext:main in my module foo is src/index.js, the following import statement will be problematic: import something from "foo/bar/baz.js". I don't think this is a big deal, and in fact we are encouraging people to define the api of the module (exporting whatever is relevant) in the main and jsnext:main, and keep the guts of the package as internal to avoid introducing a refactor hazard, but we should probably document that somewhere.

In fact, it was my desire to use small chunks of large libraries without using the folder structure approach that led me to create Rollup - it seemed to me that the ES6 modules spec had been designed with exactly this in mind, and we just needed tooling that was designed around that idea. (@caridy - is there an authoritative answer to this question, or is it up to the community still to reach a consensus?)

but the point is to see them as two sides of the same coin as opposed to competing solutions. I would love to be able to offer this option out of the box in jspm

Amen! I think there's huge value in well-integrated solutions - the success of npm+Browserify is testament to that. Perhaps jspm static-bundle would be a more logical place to try and implement this stuff than inside Rollup itself - maybe we should try and figure out how we'd go about doing that?

Out of interest - have you looked at the optimizations made in the bundling of ES6-Module-Transpiler?

Yes, in fact I feel like I'm continually trying to catch up with @eventualbuddha! We both contribute to Esperanto, which works similarly to es6-m-t. Rollup is sort of a sequel to Esperanto (I decided it was better off as a separate project, at least at first, so I could develop without being weighed down by the mistakes I made in the design of Esperanto) - both Rollup and Esperanto owe a lot to es6-m-t, not least the test suite.

[guybedford]

@Rich-Harris I'm sure a jspm resolver could be written through that easily. So resolveExternal can be provided via an option? SystemJS now provides an normalizeSync function on its interface, so the jspm API could easily be piped into this via that most likely. Would be happy to help prototype some work like this. Then in due course we could even say that SFX bundles in jspm default to using a library like this when all modules in the tree are ES6 for example. Would be nice to have this as an "optimization" to the system.

So the idea of entry points as exports only works in the world of tree shaking, which only works for the scenarios this project considers. In the dynamic loader in the browser, there is no concept of tree shaking, as a module is a module regardless of which exports are used. In these cases, our only alternative to tree shaking is the module's public file interface to load just what we need. When I say two sides of the same coin, it helps to be aware of the benefits and needs of these two approaches.

If you're playing catch-up to Brian, you're following in good foot steps!

[Rich-Harris]

Cool thanks, I will look into normalizeSync (though I should have mentioned - resolveExternal can return a string or a promise).

In the dynamic loader in the browser, there is no concept of tree shaking, as a module is a module regardless of which exports are used

Good point, I hadn't considered that scenario. Though you can still get some of the benefits - to use the same example as above, if you had import { ascending } from 'd3' in your app, then a loader that worked along similar lines to Rollup could load D3's main file, figure out where ascending was defined, and only make requests for that file and its dependencies. At most it's one extra HTTP request (for the main file) and the cost of some unused code (which is likely to be small) in the modules that get loaded for the bits we do need.

[guybedford]

Interesting, so you're saying the index.js can define the entry points along the lines of:

d3/index.js

export { ascending } from './ascending.js';
export { another } from './another.js';

I'm not sure what benefits we get though by defining these entry points via a JS convention (which breaks down as soon as users add executing code to that index file) over just documenting a public path-based API.

[Rich-Harris]

Exactly - the takeaway I had from the conversation quoted above was that this was to be the preferred way of defining an interface with ES6 modules. At first I was highly sceptical and thought that a filesystem-based approach was the only pragmatic way to do it, but after stewing on it a while I've crossed the fence.

I actually had a go at doing exactly this with D3 - see d3-jsnext, and specifically the main file. (It's obsolete now that D3 is moving to ES6.) When time allows I want to apply a similar treatment to other large libraries like lodash and three.js.

I'm not sure what benefits we get though by defining these entry points via a JS convention

From the consumer's point of view, I think it makes for more learnable APIs, because there's no compromise involved (what makes sense for code organisation doesn't always make sense for the API). For example is D3's voronoi layout in d3/src/layout/voronoi or d3/src/geom/voronoi? import {voronoi} is much easier to remember. Similarly, import {pluck} from 'lodash' is much easier than remembering that pluck lives in the collections folder.

For authors I think the benefits are more compelling - a more malleable codebase, less to document, and more flexibility over distribution (my-library.min.js served from a CDN has the same interface as the jspm install'd version).

(which breaks down as soon as users add executing code to that index file)

Maybe this is a bit pie-in-the-sky... but you don't have to execute it :) You only need to parse the code to find the relevant export statement, you don't need to run it. Although I'm certain that introduces all manner of complexities.

[guybedford]

Yeah it's too much of an exception to standard module execution, and it's not a convention that can be assumed widely enough to skip standard module loading principles.

That said I completely understand the motivations.

What are the proposals looking like these days for multiple entry points being defined via configuration in the package.json? May well be a better common ground.

[Rich-Harris]

I don't know to be honest, my ear isn't that close to the ground. Can you think of anyone we should approach who might have a more definitive answer to these questions?

[guybedford]

Sure, personally I'm more inclined to look for the right place to have the open discussions. It may still also be a little early for this one, but will certainly copy you in if I notice it come up again anywhere.

[Rich-Harris]

Sounds good, and likewise - will cc you if I see anything.

[callumlocke]

Just been catching up with this discussion... two things

• I can't find resolvePath (mentioned above)... has this been renamed resolveId?
• it looks like resolveExternal option still exists, but is undocumented on the API page. Is this deprecated?

[Victorystick]

@callumlocke Yes, it's been renamed resolveId, and resolveExternal is documented on the API page.

[callumlocke]

So it is, sorry!

[Rich-Harris]

Closing this issue – custom resolvers/loaders/transformers can now be implemented fairly straightforwardly as plugins. If necessary we could create a rollup-plugin-jspm plugin similar to rollup-plugin-npm.

[jay8t6]

@Rich-Harris any plans on creating a plugin for jspm, rollup-plugin-jspm? I am trying to load the commonjs modules via jspm and running out of options.

[Rich-Harris]

@jay8t6 it would be a worthwhile addition – personally I don't have a need for it, and have a million other things I need to do first, so I'll defer to anyone else out there who wants to do it! https://github.com/rollup/rollup-plugin-npm ought to be a good place to start from.

[jay8t6]

@guybedford would I be able to use the jspm config.js file content in resolveId function? to map external dependencies?

[Victorystick]

@jay8t6 Sure! Make your custom resolveId function call an implementation of SystemJS's module resolution logic. If it works well, you could always make it an NPM package, and share it with the community. (We haven't gotten around to it yet, but if enough people request it, we'll create one.)

[guybedford]

@jay8t6 yes jspm provides the normalization API via https://github.com/jspm/jspm-cli/blob/master/docs/api.md#normalizename-parentname---promisenormalized, although it is promise based, so you may be better off using a loader instance against a loader instance normalizeSync call.