Fix production build and default to it

[tibdex]

References

• Feature request: Disable inline sourcemaps on dev builds #6804 (comment)
• webpack --config webpack.prod.config.js hanging  #4276

Code changes

The changes are small an explained in comments and in the commit messages.

User-facing changes

End-users will have a better UX since they will be running optimized code.

Backwards-incompatible changes

There should be none as switching to prod build by default is non breaking and Webpack is an implementation detail.

[jupyterlab-dev-mode]

Thanks for making a pull request to JupyterLab!

To try out this branch on binder, follow this link:

[telamonian]

Personally, I think this is a fine idea. However, the decision to make jupyter lab build a dev mode build was made before I got deeply involved in build system issues.

Can some who does know why jupyter lab build was switch to dev mode please weigh in on this PR? Maybe one of @blink1073 @ian-r-rose @gnestor?

[ian-r-rose]

It was switched to default to dev mode for not-particularly-great reasons. Primarily, dev mode requires less resources and has shorter build times. This was particularly relevant to repos used in mybinder.org, which would often crash due to hitting RAM limits. The alternative would be to communicate very well to binder users and developers that they may want to build in dev mode.

[telamonian]

@ian-r-rose Okay, so currently for everyone (extension devs, binder users, end-users who install an extension) we're always doing dev mode builds, right? Is there any category of user that would benefit from the production Webpack mode?

If not, then there's a clear solution: we default jupyter lab build to a Jupyterlab production build, but we still set the Wepack mode to development in jupyterlab/staging/webpack.prod.config.js.

In terms of naming this is somewhat convoluted (ie a production Jupyterlab build would trigger a development Webpack build), but that complexity should only impact the poor souls maintaining the build system.

[afshin]

Pre-1.0 jupyter lab build defaulted to dev_build being True, but I think it's a reasonable thing to say that now that there's a stable major version, jupyter lab build defaults to production mode and we have to pass in the flag if we prefer to be in development mode.

[tibdex]

Primarily, dev mode requires less resources and has shorter build times. This was particularly relevant to repos used in mybinder.org, which would often crash due to hitting RAM limits.

With minimization disabled (what this PR does), I can't see any difference in build time between dev and prod mode on my laptop. They both take around 13s.

Is there any category of user that would benefit from the production Webpack mode?

End-users would greatly benefit from Webpack production mode since it defines the environment variable NODE_ENV=production and it's what Webpack uses to bundle either the development or production optimized build of packages such as React. So, with Webpack production mode, end-users would be running more efficient code and thus would have better performance and experience.

we default jupyter lab build to a Jupyterlab production build, but we still set the Wepack mode to development

That would defeat the purpose of the production mode explained above.

I think it's a reasonable thing to say that now that there's a stable major version, jupyter lab build defaults to production mode and we have to pass in the flag if we prefer to be in development mode.

👍

[vidartf]

If I remember correctly, there was also an issue that production builds:

• use more memory, so are more likely to trigger OOM exceptions
• caused some issues with long filenames on Windows?

I might be misremembering though. Please do a bit of digging in the repo and look for previous discussions. I would do it myself but I'm on mobile only for the rest of the week.

[blink1073]

Yes, the primary reason is memory. We worked around the Windows path issue: https://github.com/jupyterlab/jupyterlab/blob/master/dev_mode/webpack.prod.config.js#L23

[telamonian]

I did some tinkering and some memory profiling with the build system. Here's the most interesting results (OSX 14.4, 16 GB ram):

• The terser plugin works as expected, and can as much as halve the final bundle size (10 MB vs 20 MB).
• For a pip install jupyterlab installation, the peak memory consumption actually went down for a production build:
  • dev build: ~900 MB peak
  • production build: ~800 MB peak
• For an installation from a local copy of the jlab repository, the peak memory consumption doubles for a production build:
  • dev build: ~1 GB peak
  • production build: ~2.05 GB peak
• Linked extensions (ie extensions in development) increase the peak memory usage of all builds, whereas extensions installed as packages from npm don't have much of an effect.
• The parallel option of TerserPlugin is of questionable value. In some cases its use gives a speedup of a few seconds, while in others it causes a considerable slow-down. In the cases where parallel makes things slower, it typically also consumes a few extra 100 MB of memory.

[telamonian]

The minification provided by TerserPlugin seems to work well for my environment, at least, so I don't think we should get rid of it entirely. With the above data in mind, here's my proposal:

• We follow @tibdex's lead and make the production build the default for jupyter lab build.
• We keep the current minification code as is, but make minimize an option (alongside dev-build in LabBuildApp) that defaults to True. Reasons:
  • Minification seems to work well with the released code, and doesn't consume excess resources under those conditions.
  • If a build fails due to an error (possibly OOM), we can add an error message that hints the user towards the use of the --minimize=False option.
  • Developers/power users can set the default for minimize (and all of the other LabBuildApp options) via config files.

On the fancier side of things, we could also do stuff like trigger dev builds for jupyter labextension link vs production builds for jupyter labextension install. Or maybe always do dev builds if a user has at least one locally linked package (ie if they're developing a package).

[blink1073]

I like the idea of using dev as the default if there are any local packages at all. 👍 to the rest of your points @telamonian

[vidartf]

Note: It might be that the memory issues were with the old Uglify plugin, and that with the Terser plugin this is resolved.

[telamonian]

I like the idea of using dev as the default if there are any local packages at all

@blink1073 The question will be how to robustly detect that. The "linkedPackages" field is the obvious way, but as currently confiugred will it give any false positives? I'll look into it.

[blink1073]

I meant the local_extensions metadata in build_config.json. https://jupyterlab.readthedocs.io/en/stable/user/extensions.html#settings

[telamonian]

I meant the local_extensions metadata in build_config.json

As it turns out, I did end up drawing the data from build_config.json, via the .info['linked_packages'] and .info['local-extensions'] attributes of a _AppHandler object:

jupyterlab/jupyterlab/commands.py

Lines 477 to 485 in cd5e119

	class _AppHandler(object):
	def __init__(self, app_dir, logger=None, kill_event=None):
	"""Create a new _AppHandler object
	"""
	self.app_dir = app_dir or get_app_dir()
	self.sys_dir = get_app_dir()
	self.logger = _ensure_logger(logger)
	self.info = self._get_app_info()

I added a couple of commits to this PR that:

• add minimize as an option for lab build.
• defaults the lab build to dev if there's anything in the "linked_packages" or "local_extensions" fields of build_config.json, and to prod otherwise.

Unfortunately, I completely screwed up merging those commits into this PR, and I can't seem to force push any fixes. Anyone know how I can un-snafu this?

Regrettably (sorry @tibdex), it required a force push, but I fixed it.

[vidartf]

Will this actually do a dev build with minimization turned on?

[telamonian]

No, the minimize there is like a no-op: jlpm build:dev:minimize will just call jlpm build:dev. I included it

• A. to help simplify the semantics of handling multiple options, especially if we ever add another one beyond build-dev and minimize
• B. as a placeholder, in case someone comes up with a good reason why a minified dev build should exist

[tibdex]

Thanks for your changes @telamonian. I agree with the new logic to decide whether to minify or not.

LGTM!

[telamonian]

@blink1073 Does the current state of the PR look good to you?

[blink1073]

After a rebase I think the logic looks good. Let’s wait until master is at 1.1dev to merge since this is a cli change

[telamonian]

Arg. My gitfu is weak when it comes to different branch names on local and remote. I managed to push my local master onto tibdex:master. Since my master is in sync with the main jlab repo, that apparently emptied all of the commits from this PR and automatically closed it. Who knew?

Since I no longer have permission to push to tibdex:master now that the PR is closed (and can't reopen it since it's empty), I'll just start a new PR

Edit

New PR at #6907

[blink1073]

Ah, yeah, maintainers can’t rebase on someone’a master branch