Resolve (external) paths when bundling

[Rich-Harris]

This discussion started on #59 (comment) but I think it's probably better considered as a separate issue.

Right now, all non-relative import paths are resolved relative to options.base (or process.cwd(), if that option is missing), and treated as external modules if they can't be found. (see also #31)

That excludes the possibility of having e.g. a separate vendor folder, or bundling modules found in node_modules. (True, those external dependencies typically won't be ES6 modules, and Esperanto only knows how to read ES6 modules. But @caridy mentions some valid use cases.)

There are two (possibly complementary) solutions:

• a resolvePath function
• a RequireJS-style paths config, where module IDs are mapped to paths (relative to base, or absolute):

esperanto.bundle({
  base: 'src',
  entry: 'main',
  paths: {
    foo: '../bower_components/foo/es6'
  }
});

[caridy]

Here is an example: https://github.com/yahoo/react-intl, where it depends on pkgs like intl-messageformat, which should be bundled up within react-intl browser's distro based on package.json->jsnext:main from intl-messageformat to point to the entry point of that library written in ES6.

[Rich-Harris]

@caridy I like the jsnext:main idea - is that a widely accepted convention, or purely an es6-m-t thing? Have to admit I hadn't seen it before.

[caridy]

The idea was proposed by @wycats a while ago, we have been using it in FormatJS project (all packages), and we have discussed it with the modules champions at TC39 as a way to signal an entry point for NPM packages that contains an ES6 distro. Lastly, in december we get a chance to discussed it with @othiym23, and he is OK with it, although NPM does ignores it (for now). Email me (caridy at gmail dot com) and I can share the link to the original doc.

[othiym23]

I can guarantee that npm will never use : as a sigil in any of its own property names in package.json, which means that @caridy's proposal won't get into conflict with npm. I think using a convention like that is a great way to play around with this stuff now. 👍

[mmun]

Can this also be supported in the non-bundling case?

Use case: Suppose you write a module A in ES6 that imports an external ES6-written module B, and you want to want to produce a concatenated, transpiled AMD module for all the modules in A and B (e.g. to run tests in a browser, Ember does this with its microlibs).

[Rich-Harris]

Some progress on this front - have opened PR #99. It basically works as planned:

esperanto.bundle({
  base: 'src',
  entry: 'main.js',
  resolvePath: function ( importee, importer ) {
    // return a string representing the path, or a promise
    // that resolves to same
  }
}).then( ... );

As a starting point, magic-string (one of Esperanto's dependencies) is now using resolvePath to bundle its dependency, vlq, which as of this afternoon uses the jsnext:main field in package.json.

The magic-string build definition includes a function that uses node-resolve to find the package and replace main with jsnext:main. There might be some value in including this function for convenience...

{
  resolvePath: esperanto.useJsnextMain // or something less ugly
}

...but for now it's a manual process. Will probably try this out in a few more places before merging the PR, but if anyone has feedback in the meantime then shout.

@mmun I'm not sure you mean by supporting this in the non-bundling case - if external libs are involved aren't we bundling by definition? Could you elaborate?

[caridy]

awesome!

Definitely not useJsnextMain, that name is ugly, jejejeje. maybe adding resolver somewhere.

One note about jsnext:main that I forgot to mention before :), it does not support specifying paths within a module, saying, if the value of jsnext:main in my module foo is src/index.js, the following import statement will be problematic: import something from "foo/bar/baz.js". I don't think this is a big deal, and in fact we are encouraging people to define the api of the module (exporting whatever is relevant) in the main and jsnext:main, and keep the guts of the package as internal to avoid introducing a refactor hazard, but we should probably document that somewhere.

[trek]

@mmun @Rich-Harris: this use case is easy to explain if you compare a particular workflow to node/npm.

Say you have a two libraries a and b, each published to npm, with source hosted on github. b depends on a. A change you'd like to make in b requires a change to a. In node you can

git clone git://a.git
git clone git://b.git

cd b
npm link ../a

And now make changes in b for the hypothetical time when the changes you're simultaneously making in a are also published.

This is easy in node because of shared module system and resolver.

In Emberland, at least, this workflow is vary painful. We author pretty much everything in ES6 and the platform as a whole moves quickly, requiring updates to many smaller libraries at once. There are other ways to get this workflow working, but they fail the Just Works™ test.

[Rich-Harris]

Further update: Esperanto itself is now using this to bundle magic-string and vlq as part of the browser build (which basically exists for the sole purpose of esperantojs.org - see esperanto.browser.js). Going through this dogfooding process uncovered one bug, and served to demonstrate that it makes much more sense to use relative paths for internal modules (one of magic-string's internal files referred to utils/encode rather than ../utils/encode, which obviously can't be resolved correctly against the base directory of its dependent).

Other than that, it seems to work pretty well - though it's not (and is not intended to be) a replacement for browserify/webpack/jspm, since it doesn't do various things like deduplication. Maybe that will become important in the future, though I don't see it as an immediate priority.

@caridy

it does not support specifying paths within a module

Thanks for explaining that, it's relevant to something I'm hoping to get round to shortly. We use d3 from time to time, and it's a fairly big library. Since we only ever use a tiny part of it, it would be good to have an ES6 fork of it and replace this...

import d3 from 'd3';
var layout = d3.layout.force()...

...with...

import forceLayout from 'd3/layout/force';
var layout = forceLayout()...

...but given what you say, that's probably not the best approach, and we need to do something more like this:

// d3/index.js
import layout_force from './layout/force';
import geo_projection from './geo/projection'; // etc
...

export { layout_force, geo_projection, ... };

// app.js
import { layout_force } from 'd3';

Have I understood that correctly? I'm not totally sure how I feel about it - I agree with the point about refactoring hazard, but it does presumably mean that all the other imports in d3/index.js get followed unnecessarily, and we have to be careful to eliminate that dead code in a later step. It also means that libraries like d3 (or lodash/amp/whatever) that are essentially collections of small modules can't effectively group their exports without using a naming convention to fake a namespace (i.e. export { someNamespace_someName };). Not at all sure what the solution is though, and perhaps the discussion is better had separately.

@mmun @trek Ah, I think I understand, the situation you want is this: building b causes b/src/b.js to be transpiled to b/amd/b.js, but also causes a/src/a.js to be transpiled to b/amd/a.js because b depends on a, and that should happen automatically, with no special configuration to deal with a other than the fact that it's been npm linked. Have I understood that correctly? If so, it makes sense, I can see why that would be painful at the moment.

[mmun]

@Rich-Harris Yes, exactly.

Re: import forceLayout from 'd3/layout/force';

The reason this breaks down in Node is that paths are always referenced relative to the root of the module and not relative to the module's main.

For example, Handlebar's main is lib/index.js. Using the ES6 source, you might have an import like

import Visitor from "handlebars/compiler/visitor";

This would fail in Node if directly transpiled to require("handlebars/compiler/visitor") because there is no file at "handlebars/compiler/visitor". The transpiler would need to be aware of the main path and inject it: require("handlebars/lib/compiler/visitor").

Like Caridy said, small modules with flat exports are generally better. That said, not not all existing ES6 code is structured that way and there are arguments in favor of paths (import forceLayout from 'd3/layout/force'; seems reasonable?).

[caridy]

Have I understood that correctly?

Yes, you did!

[mmun]

In short, we want to transpile node modules written in ES6 that depend on node modules written in ES6.

[trek]

@mmun which is a very different idea than I think most people have for module transpilation.

The current practice seems to assume each library author will have already transpiled (using the transpiler of their choice) from ES6 modules and/or other features into ES5 in some module flavor.

Consumers of those libraries (who are also library authors) can author in ES6, but will consume external dependencies in ES5 and rely on CJS/AMD loader semantics.

This all seems good, but there's a lot of value in letting an application developer delay transpilation until just the moment of publication and use their preferred toolchain. That's probably one of our main use cases.

[mmun]

@trek Yes, exactly. That blog post works great if you want to write your module in ES6 and compile it to CJS as a pre-publish step. It assumes that the module is intended to be used in a CJS environment (or transpiled through Browserify).

There isn't a way to build up a tree of ES6 source across several NPM packages and compile directly to, say, AMD.

[caridy]

@trek @mmun that's what we are aiming for with this. Few notes:

• libraries should be compiled so they can be used today without the hazard of having to add a transpilation step for a code that you don't write (assuming you're consuming the library in your app). An example of this will be npm install intl-messageformat-parser, which can be used with webpack/browserify, or just used on the server side by using require('intl-messageformat-parser').
• libraries source should be available as ES6 format if they are mean to be used thru a build process to facilitate code analysis, dead code elimination, sandboxing/isolation or just simply a better performance thru a folding process. E.g.: https://github.com/yahoo/ember-intl which uses intl-messageformat-parser under the hood, and can generate a bundle/fold with all its dependencies. The same applies to app level code that wants to use dependencies thru their original format.

[acusti]

This is great! I’m so glad I stumbled across it. I’ve been oh-so-painfully trying to piece together a usable workflow for publishing and consuming small and simple ES6 modules via npm that also can do the CJS/AMD/window.global dance for others, and have felt awfully sympathetic towards @trek’s “Emo Hellscape” sentiment.

I resorted to a terrible hack for one module (which only has a single dependency) where I set esperanto’s basedir option to the src directory of the dependency, and I am so relieved that I will be able to drop it.

From my perspective, having the esnext resolver convenience function available for the resolvePath option as suggested here would be super helpful. It would also be great to have that part of the resolvePath functionality exposed somehow as an option for the CLI. Perhaps as --resolve-esnext, or similar?

Thanks for this very useful project and to all for charting a way through this murky mess of mystery and madness.

[Rich-Harris]

Closing this now that Rollup is taking over from Esperanto (#191). Rollup searches node_modules for packages with jsnext:main by default, and the resolver and loader can both be overridden to support whatever other scheme is necessary.