Morph Neutrino API and CLI into middleware injectors for external CLI tools #852

eliperelman · 2018-05-07T14:21:36Z

Edited to reflect current state of PR.

Fixes #708.
Fixes #736.
Fixes #870.
Fixes #842.
Closes #773.
Closes #839.
Closes #849.

Outstanding issues to file:

Update docs to reflect these API changes (Update documentation to reflect changes in API from switching to external CLI tools #896)
~~Determine need for config-loading bins instead of forcing config file boilerplate (still unsure if we should do this because of IDE integration)~~
Modify stats to produce friendlier build output (Modify stats to produce friendlier build output #897)

This patch is a work-in-progress, and is quite the overhaul of Neutrino into something different. Right now there is no change to the documentation, which I will change if we decide to move forward with this.

This patch morphs Neutrino into a "preset engine" for transforming managed webpack configuration into a format that can be consumed by other CLI tools. No longer would we wrap external CLI tools like webpack, eslint, jest, karma, mocha, stylelint, etc. Instead, Neutrino could augment these tools by exposing methods for returning usable configuration.

This approach gives the user control of their CLI tool back, and they can programmatically invoke Neutrino to load their configuration, and override however they like:

webpack --mode production

// webpack.config.js
const neutrino = require('neutrino');

// Load with no config overrides:
module.exports = neutrino().webpack();

// Load with config overrides:
module.exports = neutrino().webpack(config => {
  // return whatever config you want
});

// Load with no config overrides using low-level API:
module.exports = neutrino().output('webpack');

// Load with config overrides using low-level API:
module.exports = neutrino().output('webpack', config => {
  // return whatever config you want
});

The same works for ESLint and friends too (except for mocha, since it has no config file we can be required from):

eslint src

// .eslintrc.js
const neutrino = require('neutrino');

// Load with no config overrides:
module.exports = neutrino().eslintrc();

// Load with config overrides:
module.exports = neutrino().eslintrc(config => {
  // return whatever config you want, like manipulating rules, etc.
});

This approach is predicated on having middleware defined in .neutrinorc.js, which shouldn't change at all.
This does do away with a bunch of CLI functionality like --inspect or --use, but those become less useful if we are just an interface to external CLI tools. We can discuss potentially bringing those back. This is now back in the form of the CLI with neutrino --inspect.
~~All environment variables are gone. We rely only on webpack's mode value.~~ We internally only use webpack's mode value, but still set process.env.NODE_ENV if not set based on the mode, or defaulted to production.
~~There is no longer a create-project tool, since we cannot force users to choose between competing tools like webpack-cli or webpack-command, and webpack-dev-server or webpack-serve.~~ The create-project tool is back, and we opt for webpack-cli and webpack-dev-server as opinions for now.

I'm sure there are other major things I am neglecting to mention. Let's discuss! I see so many benefits to this, including reduced maintenance burden and the ability to allow people to use Neutrino without dumping their existing tools.

cc: @mozilla-neutrino/core-contributors

eliperelman · 2018-05-07T14:34:08Z

Ah, one thing I forgot to mention is that you can override configuration still using neutrinorc with the webpack-chain instance. This means you can now override using whichever method you prefer:

// Use neutrinorc for deterministic configurability with
// the chainable API
module.exports = {
  use: [
    '@neutrinojs/react',
    neutrino => {
      neutrino.config.devtool('source-map');
    }
  ]
};

// Override from the configuration file of your CLI tool.
// This requires that you switch away from using the internal
// Neutrino CLI loader to making the tool use its own config file
// again, e.g.
// from:
// neutrino webpack --mode production
// to:
// webpack --mode production

// webpack.config.js
const neutrino = require('neutrino');

module.exports = neutrino().webpack(config => ({
  ...config,
  devtool: 'source-map'
});

timkelty · 2018-05-07T14:36:11Z

Hope to review this week, but I'm a big fan of moving in this direction!

SeeThruHead · 2018-05-07T14:47:28Z

Absolutely love this.

edmorley · 2018-05-07T21:42:19Z

Many thanks for working on this - as you know I'm keen to try and make it easier to use native tools with Neutrino / reduce the cost of maintenance on our side / possibly reduce the overhead of Neutrino in startup time :-)

I tested this by comparing master (though I refreshed the lockfile on master only, so webpack 4.8.0 on both) and this PR via the following steps:

From monorepo root: rm -rf node_modules packages/*/node_modules && yarn && yarn link:all
cd .. && create-project test-<name>
Give the following answers: Web -> React -> Jest -> AirBnb
cd test-<name>
Edit the generated package.json to remove neutrino and @neutrinojs/*
rm -rf node_modules yarn.lock && yarn
yarn link neutrino @neutrinojs/react @neutrinojs/jest @neutrinojs/airbnb
Run yarn build 5 times and take the quickest invocation according to time

On master I get:

$ time yarn build
yarn run v1.6.0
$ neutrino build
√ Building project completed
Hash: 644638fa61693d1f28fe
Version: webpack 4.8.0
Time: 3422ms
Built at: 2018-05-07 21:16:03
                          Asset       Size  Chunks             Chunk Names
runtime.a5d5684f37863467cd6d.js   1.04 KiB       0  [emitted]  runtime
      1.f5d8aa6c2a2a15538a95.js    106 KiB       1  [emitted]
 index.96536b326bb3b272ec66.css   18 bytes       2  [emitted]  index
  index.0705adefdc1c7579150f.js  562 bytes       2  [emitted]  index
                     index.html  565 bytes          [emitted]
Entrypoint index = runtime.a5d5684f37863467cd6d.js 1.f5d8aa6c2a2a15538a95.js index.96536b326bb3b272ec66.css index.0705adefdc1c7579150f.js
Done in 7.73s.

real    0m8.294s
user    0m0.000s
sys     0m0.015s

Trying with this PR I got:

$ yarn build
yarn run v1.6.0
$ neutrino webpack --mode production
events.js:165
      throw er; // Unhandled 'error' event
      ^

Error: spawn webpack ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:201:19)
    at onErrorNT (internal/child_process.js:379:16)

I presume this is due to needing child_process.spawn() changes for compatibility with Windows. I tried adding shell: true, which got further, but still failed with Error: Cannot find module '.neutrinorc.js', so I skipped this approach for now.

I instead created a webpack.config.js like so:

const neutrino = require('neutrino');
module.exports = neutrino().webpack();

And then tried using webpack directly:

$ time yarn webpack --mode production
yarn run v1.6.0
$ C:\Users\Ed\src\test-react-new\node_modules\.bin\webpack --mode production
Hash: 6d910272a91fdf310949
Version: webpack 4.8.0
Time: 2891ms
Built at: 2018-05-07 22:01:05
                          Asset       Size  Chunks             Chunk Names
runtime.d3f7781965753ba20f1b.js   1.04 KiB       0  [emitted]  runtime
      1.083d16c533c13131a3f5.js    106 KiB       1  [emitted]
                      index.css   18 bytes       2  [emitted]  index
  index.e329700ad1ece9219d57.js   1.04 KiB       2  [emitted]  index
                     index.html  548 bytes          [emitted]
Entrypoint index = runtime.d3f7781965753ba20f1b.js 1.083d16c533c13131a3f5.js index.css index.e329700ad1ece9219d57.js
 [6] ./src/App.css 39 bytes {2} [built]
 [7] ./src/App.jsx 1.01 KiB {2} [built]
[19] ./src/index.jsx 635 bytes {2} [built]
[20] multi ./src/index 28 bytes {2} [built]
    + 17 hidden modules
Child html-webpack-plugin for "index.html":
     1 asset
    Entrypoint undefined = index.html
    [0] (webpack)/buildin/module.js 497 bytes {0} [built]
    [1] (webpack)/buildin/global.js 489 bytes {0} [built]
        + 2 hidden modules
Child mini-css-extract-plugin ../neutrino-dev/node_modules/css-loader/index.js??ref--7-1!src/App.css:
    Entrypoint mini-css-extract-plugin = *
    [1] ../neutrino-dev/node_modules/css-loader??ref--7-1!./src/App.css 204 bytes {0} [built]
        + 1 hidden module
Done in 6.14s.

real    0m6.677s
user    0m0.000s
sys     0m0.015s

It looks like the configuration differs slightly:

index.css no longer has a hashed filename
index.js is larger
the stats output has changed

...however the build time reduced by 1.5s, which is promising (though might just be the different build options?).

Some other things I spotted:

create-project's initial lint didn't complete (the yarn webpack --mode production is after I fixed the issues)
I had problems getting the create-project's generated .eslintrc.js to work - running eslint failed with:

$ yarn eslint --ext=js,jsx,vue,ts,tsx,mjs src --fix
yarn run v1.6.0
$ C:\Users\Ed\src\test-react-new\node_modules\.bin\eslint --ext=js,jsx,vue,ts,tsx,mjs src --fix
Cannot read config file: C:\Users\Ed\src\test-react-new\.eslintrc.js
Error: Cannot find module '.neutrinorc'
Error: Cannot read config file: C:\Users\Ed\src\test-react-new\.eslintrc.js
Error: Cannot find module '.neutrinorc'
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:548:15)

when we get to the point of adding more polish to this approach, it would be good to improve the error message for people whose yarn build command still tries to run neutrino build, since it currently fails with the confusing:

$ yarn build
yarn run v1.6.0
$ neutrino build
C:\Users\Ed\src\neutrino-dev\packages\neutrino\bin\neutrino.js:16
spawn(command, [...args, ...commands[command]], {
                                    ^

TypeError: commands[command] is not iterable
    at Object.<anonymous> (C:\Users\Ed\src\neutrino-dev\packages\neutrino\bin\neutrino.js:16:37)

edmorley · 2018-05-07T22:12:27Z

This is similar to how nyc works, by calling a Neutrino CLI which just loads middleware and then spawns the given command passing in a config file.

Could this part be avoided by instead defining the package.json build script as:
webpack --mode production --config node_modules/neutrino/src/foo.js

I know passing a node_modules path isn't pretty, but given neutrino is a top-level dependency, we don't need to sorry about hoisting, right? I'm just wondering if we can get away with keeping the number of API usage permutations to a minimum, to reduce complexity/maintenance/potential for bugs.

Also, what are your thoughts on starting to use .babelrc.js with this approach too? (Or the new babel.config.js which avoids some of .babelrc*'s issues; see babel/babel#7358.) It would seem that many tools "just work" when a babelrc is preset (eg babel-jest) - so it might simplify several other presets.

edmorley · 2018-05-08T00:12:56Z

It looks like the configuration differs slightly:

index.css no longer has a hashed filename

index.js is larger

Ah the former is because packages/style-loader/index.js uses neutrino.config.mode === 'production' rather than neutrino.config.get('mode') === 'production'.

The latter is because the Jest preset's Babel settings are being applied even for .webpack(), and plugin-transform-modules-commonjs increases the output size.

eliperelman · 2018-05-08T12:33:35Z

Could this part be avoided by instead defining the package.json build script as:
webpack --mode production --config node_modules/neutrino/src/foo.js

Yes, that is certainly possible. If we are fine with a little loss in ergonomics here, we can move forward with it.

Also, what are your thoughts on starting to use .babelrc.js with this approach too?

This should be totally possible now, just need to add a babel() method.

Ah the former is because packages/style-loader/index.js uses neutrino.config.mode === 'production' rather than neutrino.config.get('mode') === 'production'.

Good catch, yay bugs!

The latter is because the Jest preset's Babel settings are being applied even for .webpack(), and plugin-transform-modules-commonjs increases the output size.

🤔 Hmm yeah, let me think about this one.

eliperelman · 2018-05-11T13:57:53Z

With my current changeset, the only way to affect configuration is via .neutrinorc.js or an override in a respective config file, e.g. webpack.config.js or .eslintrc.js. I took away the ability to do webpack node_modules/neutrino/whatever for now, so you need a webpack.config.js with:

const neutrino = require('neutrino');

module.exports = neutrino().webpack();

I think we should discuss if this needs to be a requirement, or if we should provide the scripts to pass to these external bins.

edmorley

I tested this locally with a webpack-cli build using --mode production, which worked great. It initially only tooks 15s on a warm rebuild which I tracked down to my .neutrinorc.js having a NODE_ENV === 'production' conditional for setting devtool source-map.

I think we still need to set process.env.NODE_ENV according to mode and/or whether command={jest,...} etc. I also can't decide whether our conditionals throughout should be from mode, command or NODE_ENV.

I think we might still need some inspect functionality (eg looking for argv.inspect), but perhaps our options will be clearer once that feature is upstreamed to webpack-chain.

I ran out of time to test further (need to head out for a show) - but will take another look tomorrow at dev server/eslint/....

Oh and sorry for so many "separate PR" comments -- I wouldn't bother so much but it's already +2400, -5700 hehe.

edmorley · 2018-05-12T15:27:17Z

package.json

-    "prettier": "^1.12.1"
+    "prettier": "^1.12.1",
+    "stylelint": "^8.4.0",
+    "webpack": "^4.7.0",


latest webpack is 4.8.2 (not that it matters much given tilde..)

edmorley · 2018-05-12T15:29:13Z

packages/banner/index.js

@@ -4,7 +4,7 @@ module.exports = (neutrino, options = {}) => {
  neutrino.config
    .plugin(options.pluginId || 'banner')
    .use(BannerPlugin, [{
-      banner: 'require(\'source-map-support\').install();',
+      banner: `require('source-map-support').install();`,


Could you split this change to a separate PR?

edmorley · 2018-05-12T15:31:36Z

packages/clean/index.js

-  delete options.pluginId;
-
+  Reflect.deleteProperty(options, 'paths');
+  Reflect.deleteProperty(options, 'pluginId');


Separate PR? :-)

edmorley · 2018-05-12T15:32:01Z

packages/compile-loader/index.js

@@ -1,7 +1,8 @@
 const babelMerge = require('babel-merge');

-module.exports = (neutrino, options = {}) => neutrino.config.module
-  .rule(options.ruleId || 'compile')
+module.exports = (neutrino, options = {}) => {


Separate PR? :-)

edmorley · 2018-05-12T15:32:56Z

packages/env/index.js

-module.exports = ({ config }, envs = []) => config
-  .plugin('env')
-  .use(EnvironmentPlugin, envs);
+module.exports = ({ config }, envs = []) => {


Separate PR? :-)

edmorley · 2018-05-12T15:54:15Z

packages/style-minify/index.js

@@ -10,4 +10,4 @@ module.exports = ({ config }, opts = {}) => {
  config
    .plugin(options.pluginId)
    .use(OptimizeCssPlugin, [options.plugin]);
-}
+};


Separate PR? :-)

edmorley · 2018-05-12T15:56:20Z

packages/web/index.js

@@ -252,13 +248,4 @@ module.exports = (neutrino, opts = {}) => {

      config.output.filename('[name].[chunkhash].js');
    });
-
-  neutrino.on('prerun', () => {
-    if (neutrino.config.entryPoints.has('vendor')) {


It would be good if we could add this check somewhere else - since it's a likely footgun for people upgrading that results in larger/slower builds (vs the other cases where it's just about cleaning up removed preferences).

.eslintignore

@@ -1,4 +1,6 @@
 packages/**/node_modules/
+# Remove ignore of neutrinorc since ESLint ignores dotfiles by default
+!.neutrinorc.js
 # TODO: Fix lint errors in our tests and remove this line
 packages/**/test/
 # The templates have any lint errors fixed during project creation,


edmorley · 2018-05-12T16:04:59Z

packages/airbnb-base/package.json

    "eslint-config-airbnb-base": "^12.1.0",
    "eslint-plugin-import": "^2.11.0"
  },
  "peerDependencies": {
+    "eslint": "^4.0.0",


The @neutrinojs/eslint package will already have a peerDependency on eslint, which will be inherited. Though might be clearer for users inspecting only the top-level package.json to be explicit? I can't decide. (The same also applies to eg the webpack peerDep in @neutrinojs/font-loader since file-loader has it's own peerDep etc)

Yeah, I added this since I figured it would be easier to ensure users would be notified from their top-level peer dependencies, but I am fine with removing some of them if they are bloating.

I think given we're now needing users to add a bunch of peerDeps perhaps we need to have it as clear as possible to them :-)

edmorley · 2018-05-12T16:14:32Z

packages/eslint/index.js

+    destination &&
+    destination.eslint &&
+    destination.eslint.rules
+  ) || {};
  const rules = deepmerge(sourceRules, destinationRules, {
    arrayMerge(source, destination) {
      return destination;


I can't select the line, since it's further down below - but there's a leftover neutrino.options.command !== 'start'.

I wonder if command could be renamed to something else so third-party middleware / end-users' .neutrinorc.js conditionals fail hard rather than silently not matching?

I think target was suggested at one point, e.g. neutrino.options.target, but I'm not convinced it wouldn't be confused with config.target.

edmorley · 2018-05-12T16:56:03Z

I think we might still need some inspect functionality (eg looking for argv.inspect)

Oh and the debug functionality (for me, main use I have had for it is preset-env's debug output).

edmorley · 2018-05-14T12:48:10Z

I wonder if we should also make webpack() and friends throw if mode is not set? (ie if the user forgets to --mode ...)

ringods · 2018-05-17T12:38:32Z

@eliperelman testing my multi-compiler question from #876 against this rework.

With the new setup, would I have something like this in my webpack.config.js?

const neutrino = require('neutrino');

module.exports = neutrino().use('my-preset-generating-multi-compiler')

What would the api be then for a preset to return a list of webpack configurations?
I'm not understanding it completely at the moment.

edmorley · 2018-05-17T13:02:44Z

packages/jest/src/index.js

+  neutrino.config.when(
+    neutrino.config.module.rules.has('compile') &&
+    neutrino.options.command === 'jest',
+    () => {


Given that the false case isn't used, and we're not using a passed (config), this might be cleaner as:

if ( neutrino.config.module.rules.has("compile") && neutrino.options.command === "jest" ) { // ... }

edmorley · 2018-05-17T13:06:11Z

packages/mocha/package.json

@@ -2,7 +2,7 @@
  "name": "@neutrinojs/mocha",
  "version": "8.2.0",
  "description": "Neutrino preset for testing Neutrino projects with Mocha",
-  "main": "src/index.js",


Similar to the jest preset case, could the file be left under src/ for now, so that git knows it's existing content and displays the actual diff? (Happy for the file to be moved either in a PR before this lands, or sometime after)

edmorley · 2018-05-17T13:11:33Z

packages/neutrino/karma.js

+    .command('karma')
+    .init();
+
+  if (neutrino.config.module.rules.has('compile')) {


It feels like these really belong in the preset rather than in the Neutrino package. Could we instead maybe do something like either:

make these files be mostly a shell that require()s the relevant parts from the preset

have the presets dynamically register new commands, so we don't need to hardcode each test runner in the neutrino package?

Hmm, I hadn't thought of it this way before, and it would take re-working the adapter pattern, but let me see what I can do.

eliperelman · 2018-05-17T13:28:02Z

@ringods I think the idea would be that you could export multiple Neutrino calls:

EDITED

// webpack.config.js
const neutrino = require('neutrino');

module.exports = [
  neutrino().output('webpack'),
  neutrino(middleware).output('webpack')
];

edmorley · 2018-05-17T20:59:36Z

To add to the commit message:

Fixes #708.
Fixes #736.
Fixes #870.
Closes #773.
Closes #849.

We should also check whether it fixes any of:
#842 (whilst Neutrino test won't exist, we should check for say .mocha() when no mocha preset)
#839
#382 (though I suspect this one will need extra changes; notably shifting to making eslint-loader use eslintrc as the source of truth)

eliperelman · 2018-05-17T23:05:44Z

Updated description to reflect removal of adapter pattern, and now using:

neutrino().output(name, override);

neutrino().output('webpack');
neutrino().output('webpack', config => {
  return config;
});

edmorley · 2018-05-17T23:51:23Z

packages/web/test/web_test.js


  const err = t.throws(() => api.use(require('..'), { minify: { image: true } }));
  t.true(err.message.includes('The minify.image option has been removed'));
 });
-
-test('throws when vendor entrypoint defined', async t => {


Could you re-add this test? (though it will need adjusting)

I moved this test to inside neutrino since this wasn't web's concern.

edmorley · 2018-05-17T23:52:44Z

packages/web/index.js

@@ -92,7 +93,6 @@ module.exports = (neutrino, opts = {}) => {
      ],
      presets: [
        [require.resolve('@babel/preset-env'), {
-          debug: neutrino.options.debug,


Could you re-add this?

edmorley · 2018-05-18T00:01:55Z

packages/react/package.json

@@ -31,7 +31,6 @@
    "@neutrinojs/loader-merge": "^8.2.0",
    "@neutrinojs/web": "^8.2.0",
    "deepmerge": "^1.5.2",
-    "eslint": "^4.19.1",


So I guess now if people don't add eslint to their package.json they will get peer dependency warnings from eslint-plugin-react (and similar for the vue/preact presets etc). However maybe that's ok and their choice to save the dependency?

The only option I can think of (other than leaving this dependency in here), is to stop the react preset from including the eslint react plugin, and then airbnb guide users can just use the airbnb preset (which does include the plugin), and for standardjs perhaps we'd have to add a second "standardjs-react" preset? (Maybe not worth it...)

Hmm yeah, hadn't thought of this one. I think the easiest thing at the moment would be to leave it until we think of something better, if possible.

edmorley · 2018-05-18T00:02:31Z

packages/react-components/test/components_test.js

@@ -0,0 +1,51 @@
+import test from 'ava';


Awesome for the extra tests!

edmorley · 2018-05-18T00:11:09Z

packages/neutrino/test/api_test.js


  t.notDeepEqual(api.config.toConfig(), {});
 });

-test('throws when trying to call() a non-registered command', t => {


Do you think we should turn this into a test checking for the Unable to find an output handler named behaviour?

Added this to the new package_test.

edmorley · 2018-05-18T00:15:00Z

packages/neutrino/package.json

@@ -2,16 +2,11 @@
  "name": "neutrino",
  "version": "8.2.0",
  "description": "Create and build JS applications with managed configurations",
-  "main": "src/index.js",
-  "bin": {
-    "neutrino": "./bin/neutrino.js"


Do you think we should leave a bin script that outputs something like "The neutrino command has been removed. See migration guide"?

Works for me.

edmorley · 2018-05-18T00:22:33Z

packages/karma/package.json

-    "karma-coverage": "^1.1.2",
-    "karma-mocha": "^1.3.0",
-    "karma-mocha-reporter": "^2.2.5",
-    "karma-webpack": "^3.0.0",


I think some of these are still needed?

I have moved them to peer deps, although I'm not entirely happy about that.

edmorley · 2018-05-18T00:26:22Z

packages/standardjs/test/standardjs_test.js

-});
-
-test('exposes eslintrc config', t => {
-  t.is(typeof Neutrino().use(mw()).call('eslintrc'), 'object');


Should we convert this and the other "expose XYZ" to checking that ...output('...') works?

edmorley · 2018-05-18T00:27:22Z

package.json

@@ -34,17 +33,21 @@
  },
  "devDependencies": {
    "auto-changelog": "^1.4.6",
-    "ava": "^0.25.0",
+    "ava": "1.0.0-beta.4",


Sneaking in more changes again hehe.. :-)

Rebase artifact. I'll fix. 😄

edmorley · 2018-05-18T00:30:26Z

packages/create-project/bin/create-project.js

@@ -22,5 +23,6 @@ const name = basename(directory);
 env.run('create-project', {
  directory,
  name,
+  registry: cli.registry,


Is this change so that the global yarn registry setting doesn't need to be adjusted? (And so can be removed from the script.) If so, this would be good since means running these tests don't affect global state locally.

In theory, yes. You can either pass --registry during create-project, or you can specify the registry in the test generator (see the project() function in test/cli_test).

It sounds like this is unrelated to the other changes here, and so belongs in a separate PR?
I'll open one for some of these cleanup things now.

eliperelman · 2018-05-18T01:46:36Z

OK, I think have the latest round of review comments addressed.

edmorley

Posting this so you don't have to wait for me to finish up the rest of the testing etc first.

It occurred to me for could always replace the .output('name') API with .webpack() et al, by making use of ES6 Proxy traps to catch unknown commands and handle them gracefully? (Which I presume is the reason why the switch to .output()?). Either way works for me though, so happy to go with your choice :-)

edmorley · 2018-05-18T11:41:10Z

packages/compile-loader/package.json

@@ -25,8 +25,7 @@
  "dependencies": {
    "@babel/core": "^7.0.0-beta.46",
    "babel-loader": "^8.0.0-beta.2",
-    "babel-merge": "^1.1.1",
-    "webpack": "^4.7.0"


Should we move this to peerDependencies rather than removing it?
eg @neutrinojs/clean kept is as a peer dep even though index.js doesn't refer to it.

Perhaps it might just be easier to add the webpack peer dep to every single Neutrino package to save having to think about it? (And to make it 200% clear to people using Neutrino what version of webpack we support.) All presets have to be used with Neutrino, which already has a peer dep on webpack, so won't really affect whether people see warnings or not. (Though the jest preset doesn't technically require it I suppose..)

Some people have voiced their want to use Neutrino middleware to test or lint things without actually building, so I would want to only provide it where necessary, if possible.

I think the reason clean keeps it is because the webpack plugin also peerDeps on webpack.

In this case babel-loader has a peer dep on webpack, so if we're going to explicitly mirror the deps peer deps, we should add it to compile-loader too.

edmorley · 2018-05-18T11:45:44Z

packages/create-project/bin/create-project.js

@@ -22,5 +23,6 @@ const name = basename(directory);
 env.run('create-project', {
  directory,
  name,
+  registry: cli.registry,


It sounds like this is unrelated to the other changes here, and so belongs in a separate PR?
I'll open one for some of these cleanup things now.

… tools Remove all CLIs, remove create project, expose command name from each config method Fix test references to API Simplify adapter pattern for overriding tool options More webpack peerDeps changes Remove irrelevant changes from patch Reset jest source files location, upgrade to latest webpack Fail lint if mode is production Add support for options.debug and options.inspect Throw if vendor entrypoint exists Re-work create-project to support external CLIs and test with verdaccio Fix bugs in beta verdaccio setup Remove localhost yarn.lock entries :/ Trying single-instance tests again in CI Removing unused environment variable detection in tests Add back karma and mocha middleware for eslint settings Experiment with using ramdisk for yarn cache Use updated yarn cache directory Update yarn.lock after rebase Remove adapter pattern, move output handlers to middleware, parse --mode from argv Fix output command for create-project mocha Removing rebase artifacts Use babel register from mocha output options Remove reference to old yarn cache script Add tests for exposing commands, add peer deps for karma Revert back to previous ava Use a proxy to expose output handlers as named methods on neutrino package Revert changes to eslintignore and create-project that were out of scope

edmorley

Last few things and hopefully this should be good to go! :-)

edmorley · 2018-05-19T00:38:02Z

packages/compile-loader/package.json

@@ -25,8 +25,7 @@
  "dependencies": {
    "@babel/core": "^7.0.0-beta.46",
    "babel-loader": "^8.0.0-beta.2",
-    "babel-merge": "^1.1.1",
-    "webpack": "^4.7.0"


In this case babel-loader has a peer dep on webpack, so if we're going to explicitly mirror the deps peer deps, we should add it to compile-loader too.

edmorley · 2018-05-19T00:40:34Z

packages/eslint/package.json

  },
  "peerDependencies": {
+    "eslint": "^4.0.0",


eslint-loader has a peer dep on webpack, so webpack needs adding here, if we're following that pattern. Similarly, should we then add it to @neutrinojs/airbnb and @neutrinojs/airbnb-base to, since they depend on @neutrinojs/eslint?

Yeah, I guess so.

eliperelman · 2018-05-19T02:59:06Z

packages/jest/package.json

  },
  "peerDependencies": {
+    "jest": "^22.0.0",


Hmm, this one feels kinda wrong since the addition of the linting is optional based on existing lint rule. Thoughts on leaving this one out?

Oh yeah sorry we do the same for the react preset etc, so ignore this

edmorley · 2018-05-19T00:49:50Z

packages/jest/package.json

@@ -22,18 +22,16 @@
    "yarn": ">=1.2.1"
  },
  "dependencies": {
-    "@babel/core": "^7.0.0-beta.46",
+    "@babel/core": "^7.0.0-beta.47",


This version and the one in the mocha preset is now out of sync with the others in the repo. Could you revert the changes to the @babel/* deps? (At some point in the future I'm sure we'll hand bump again, but this is in-range and it was only done recently in #848.)

Sure thing, this was from moving the .jest() command from neutrino package to jest.

edmorley · 2018-05-19T00:51:41Z

packages/neutrino/bin/neutrino.js

@@ -1,130 +1,9 @@
-#!/usr/bin/env node
+#! /usr/bin/env node


Stray whitespace here (shebangs normally don't have additional whitespace before the interpreter; not all POSIX implementations support it)

edmorley · 2018-05-19T01:57:19Z

packages/karma/package.json

-    "neutrino": "^8.0.0"
+    "karma": "^2.0.0",
+    "karma-cli": "^1.0.0",
+    "mocha": "^5.0.0",


I'm presuming mocha can't be moved to dependencies due to something like the karma frameworks pref not accepting absolute depdency paths?

It was mostly to enable usage of karma and mocha presets in the same repo.

edmorley · 2018-05-19T01:59:10Z

packages/mocha/package.json

    "@neutrinojs/loader-merge": "^8.2.0",
-    "change-case": "^3.0.2",
-    "mocha": "^5.1.1",


For this preset, should mocha now be listed in peerDependencies, rather than removed?

edmorley · 2018-05-19T02:19:20Z

packages/create-project/commands/init/index.js

-      scripts.start = `${packages.NEUTRINO} start`;
+      scripts.start = this.data.project === '@neutrinojs/node'
+        ? 'webpack --watch --mode development'
+        : 'webpack-serve --mode development';


Running yarn start on a create-project with options React+Jest+AirBnb, I get:

$ yarn start yarn run v1.6.0 (node:6040) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead. $ webpack-serve --mode development × ｢config｣: An error ocurred while trying to find a config file Did you forget to specify a --require? C:\Users\Ed\src\test-project\node_modules\@webpack-contrib\config-loader\lib\load.js:84 throw new LoadConfigError(e, configPath); ^ LoadConfigError: Cannot find module 'webpack-dev-server/client' at Function.Module._resolveFilename (internal/modules/cjs/loader.js:571:15) at Function.require.resolve (C:\Users\Ed\src\test-project\node_modules\webpack-serve\node_modules\v8-compile-cache\v8-compile-cache.js:162:23) at config.entry.batch.entry (C:\Users\Ed\src\test-project\node_modules\@neutrinojs\web\index.js:235:43)

I guess we have no coverage of this at the moment?

Perhaps in test_cli.js we can do either (or both) of:

test webpack build using --mode development

see if we can spawn the dev server via yarn start, then if it starts successfully, kill it after a few seconds

Until then perhaps we need to hand-test (using create-project and --registry) all of the top-level preset's yarn start commands?

Also, I think the reason the unit tests didn't catch this is because the repo root package.json contains webpack-dev-server (when I think it perhaps shouldn't?).

Oooooooh, this is a problem. I don't think we can use webpack-serve out-of-the-box because of our hardcoded entries in web for dev server:

https://github.com/mozilla-neutrino/neutrino-dev/blob/master/packages/web/index.js#L235

Perhaps we should switch back to webpack-dev-server for now (also reduces the amount changed in one PR) and we can figure it out later?

You read my mind.

Perhaps in test_cli.js we can do either (or both) of:

test webpack build using --mode development

see if we can spawn the dev server via yarn start, then if it starts successfully, kill it after a few seconds

I've filed #886 for this.

edmorley · 2018-05-19T02:30:29Z

packages/neutrino/src/devServer.js

-
-// devServer :: (Object config -> Object api) -> Future () Function
-const devServer = (config, { options }) => Future
-  .of(merge({ devServer: { host: 'localhost', port: 5000, noInfo: !options.debug } }, config))


Does the noInfo: !options.debug need to move to packages/dev-server/index.js now? Or are we leaving it to the user to pass the options to webpack-serve?

The latter seems to make sense, I was just wondering if with the current settings things are going to get verbose. I think after this PR we'll probably want to revisit our customisation of stats entirely, so perhaps this can wait until then?

Yeah let's follow up with stats changes.

edmorley · 2018-05-19T02:42:44Z

packages/neutrino/bin/neutrino.js

-  .option('debug', {
-    description: 'Run in debug mode',
-    boolean: true,
-    default: false,


With this PR neutrino.options.debug now defaults to undefined rather than false, which means clean-webpack-plugin now gives verbose output (its default) when it didn't before. I think we need to make it default to false again.

On that note, process.env.NODE_ENV is also undefined when it wasn't before. I think we need to set that explicitly again too, per:
#852 (review)

…ncies

edmorley · 2018-05-23T23:55:08Z

The Netlify build on master is failing with:
File not found: /opt/build/repo/docs/packages/fork.md
(https://app.netlify.com/sites/neutrinojs/deploys/5b05c3691f12b73cb85e935d)

Could you update mkdocs.yml for the fork preset removal?

The `fork` preset was removed by #852, but it still referenced by the newly added mkdocs configuration.

This restores the webpack default of not outputting the full stack trace for compile errors, since it's typically pretty spammy and doesn't really help with understanding the error. Fixes #875 for dev-server (#852 already fixed it for `yarn build`).

This adjusts the console output to be less verbose, for parity with how it was prior to the switch to native CLIs in #852. In some cases the output has been made to be even more concise than in Neutrino 8, since it's now possible for users to adjust the output via CLI flags, therefore less important to satisfy all use-cases out of the box. For that reason the `debug` handling has also been removed. The output when using webpack-dev-server now includes the duration, which makes it much easier to compare incremental build times. The `performance.hints(false)` of `@neutrinojs/node` has been removed since it's redundant as of webpack 4.2.0: webpack/webpack@c65fb74 Fixes #897.

It stopped being used as of #852. Closes #1096.

) - Improve documentation based in feedback in #1129. - Add checks for the legacy Neutrino `options.host` and `options.port`, since they were removed in #852. Refs: #1129 (comment) #1129 (comment) #814 (comment)

eliperelman requested a review from edmorley May 11, 2018 13:54

edmorley mentioned this pull request May 12, 2018

Setting 'config.stats' to a named preset doesn't work #870

Closed

eliperelman mentioned this pull request May 12, 2018

Document that adding webpack to package.json is not recommended #849

Closed

edmorley reviewed May 12, 2018

View reviewed changes

This was referenced May 13, 2018

Build a Neutrino-backed version of create-react-app #556

Closed

Sort create-project dependencies and devDependencies #867

Merged

eliperelman mentioned this pull request May 17, 2018

[fork] return a multi-compiler setup rather than fork behaviour. #876

Closed

edmorley reviewed May 17, 2018

View reviewed changes

edmorley reviewed May 18, 2018

View reviewed changes

edmorley mentioned this pull request May 18, 2018

Massive error message in development when unable to resolve a file #875

Closed

edmorley reviewed May 18, 2018

View reviewed changes

edmorley mentioned this pull request May 18, 2018

Misc monorepo testing improvements #884

Merged

eliperelman added 2 commits May 18, 2018 15:43

Add back karma plugins to preset

ae11719

edmorley suggested changes May 19, 2018

View reviewed changes

The delightful adventures of yarn workspaces and monorepo dev depende…

1ab5f4e

…ncies

edmorley approved these changes May 21, 2018

View reviewed changes

This was referenced May 22, 2018

Update documentation to reflect changes in API from switching to external CLI tools #896

Closed

Modify stats to produce friendlier build output #897

Closed

eliperelman merged commit cea73d6 into neutrinojs:master May 23, 2018

edmorley mentioned this pull request May 23, 2018

Exit with non-zero exit code with message when no registered test #868

Closed

eliperelman mentioned this pull request May 23, 2018

Web Extension Preset #831

Closed

edmorley mentioned this pull request May 24, 2018

Fix the Netlify build after #852 #904

Merged

edmorley added a commit that referenced this pull request May 24, 2018

Fix the Netlify build after #852 (#904)

1bc18a8

The `fork` preset was removed by #852, but it still referenced by the newly added mkdocs configuration.

This was referenced May 24, 2018

Environment-specific configs silently not working after #852 #906

Closed

log4js can't get output. #894

Closed

This was referenced Jun 6, 2018

[v8.x] Correct the URL used for the devServer 'open' feature #933

Merged

Trace warnings in debug mode #619

Closed

dev-server: Suppress unhelpful stack trace for compile failures #936

Merged

edmorley mentioned this pull request Jun 7, 2018

Write migration guide for Neutrino 9 #944

Closed

This was referenced Jun 30, 2018

Template .eslintrc.js files no longer set 'root' #973

Closed

lint command, eslint and additional linters #612

Closed

edmorley mentioned this pull request Jul 16, 2018

Jest preset improvements #851

Closed

This was referenced Aug 31, 2018

Should @neutrinojs/karma empty the default webpack entry point? #634

Closed

Reduce verbosity of webpack compile output #1072

Merged

edmorley mentioned this pull request Sep 13, 2018

eslint: Remove unused dependency on 'debug' #1097

Merged

edmorley added a commit that referenced this pull request Sep 13, 2018

eslint: Remove unused dependency on 'debug' (#1097)

f5a97db

It stopped being used as of #852. Closes #1096.

edmorley mentioned this pull request Oct 18, 2018

Issues with @neutrinojs/eslint #1181

Closed

edmorley mentioned this pull request Dec 20, 2018

@neutrinojs/library externals cause Karma "Uncaught ReferenceError: require is not defined" #1251

Closed

This was referenced Mar 24, 2019

Neutrino 9 beta/release candidate feedback #1129

Closed

Improve migration documentation and add more legacy option checks #1338

Merged

Morph Neutrino API and CLI into middleware injectors for external CLI tools #852

Morph Neutrino API and CLI into middleware injectors for external CLI tools #852

Conversation

eliperelman commented May 7, 2018 • edited

eliperelman commented May 7, 2018

timkelty commented May 7, 2018

SeeThruHead commented May 7, 2018

edmorley commented May 7, 2018 • edited

edmorley commented May 7, 2018

edmorley commented May 8, 2018

eliperelman commented May 8, 2018

eliperelman commented May 11, 2018 • edited

edmorley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmorley commented May 12, 2018

edmorley commented May 14, 2018

ringods commented May 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eliperelman commented May 17, 2018 • edited

edmorley commented May 17, 2018

eliperelman commented May 17, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eliperelman commented May 18, 2018

edmorley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmorley left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmorley May 19, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as outdated.

eliperelman May 19, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edmorley May 19, 2018 • edited

Choose a reason for hiding this comment

eliperelman commented May 7, 2018 •

edited

edmorley commented May 7, 2018 •

edited

eliperelman commented May 11, 2018 •

edited

eliperelman commented May 17, 2018 •

edited

edmorley May 19, 2018 •

edited

eliperelman May 19, 2018 •

edited

edmorley May 19, 2018 •

edited

edmorley May 19, 2018 •

edited

eliperelman May 19, 2018 •

edited