2.0 release plans

[jasongrout]

This issue is about the 2.0 release. We are currently targeting January 2020 for the release.

• 18 Sep 2019: Branch 1.x off of master and publish 2.0 alpha 0
• grep the code base / issues for TODOs for 2.0 and deprecated things to remove. - see TODO/Deprecations for 2.0 #7720 for a list
• Dec 11: target date for an RC and code freeze
• Changelog: @blink1073
  • update cell tools screenshot now that the icon has changed
  • make sure the screenshot has been crushed with pngquant, following the guidelines in https://jupyterlab.readthedocs.io/en/stable/developer/contributing.html#screenshots-and-animations (looks like the cell tools has been, but the other two screenshots in that changelog directory have not, so should be)
• Extension migration guide: @afshin, @telamonian
• UX polish: @tgeorgeux
• Blog post 2.0 announcement
• Update the API docs: http://jupyterlab.github.io/jupyterlab/

Further checklist items from release.md:

• Modify and run python scripts/milestone_check.py to check the issues assigned to this milestone
• Write release highlights, starting with:

  loghub jupyterlab/jupyterlab -m XXX -t $GITHUB_TOKEN --template scripts/release_template.txt

• Test the release candidate in a clean environment
• Make sure the CI builds pass
  • The build will fail if we publish a new package because by default it is
    private. Use npm access public @jupyterlab/<name> to make it public.
  • The build will fail if we forget to include style/ in the files:
    of a package (it will fail on the jupyter lab build command because
    webpack cannot find the referenced styles to import.
• Update the other repos:
  • https://github.com/jupyterlab/extension-cookiecutter-js
  • https://github.com/jupyterlab/extension-cookiecutter-ts
  • https://github.com/jupyterlab/mimerender-cookiecutter-ts
  • https://github.com/jupyterlab/theme-cookiecutter
  • https://github.com/jupyterlab/jupyter-renderers - Updates for jlab 2.0 jupyter-renderers#216
• Add a tag to ts cookiecutter with the new JupyterLab version
• Update the extension examples:
  • Notebook toolbar button
• Update the extension tutorial
• At this point, there may have been some more commits merged. Run python scripts/milestone_check.py to check the issues assigned to this milestone one more time. Update changelog if necessary.

[telamonian]

Do we want to maybe start experimenting now with Yarn 2.0? There's a bunch of new helpful features that would let us upgrade the build system in various way (many of which are possibly breaking, so getting them into Jlab 2.0 seems appropriate).

Yarn 2.0 is currently in development over at yarnpkg/berry. It's a bit vague, but my sense is that they're planning for a release by the end of 2019. I know that at least some of the Jlab-relevent features have already been added, so it might be wise to start playing around with it now.

@blink1073 What's your take? Does experimenting with Yarn 2.0 now seem like it would be worthwhile, or is it still too early?

[blink1073]

I think it is worth trying now

[telamonian]

@blink1073 How is the yarn.js we distribute within the Jlab python package prepared?

[jasongrout]

Can we experiment without committing? I'd hate to have to delay 2.0 because we were waiting on yarn.

Here are the previous updates: https://github.com/jupyterlab/jupyterlab/commits/58a301a86aac260aa17c7d21cabdfbfcced90185/jupyterlab/staging/yarn.js

We just copy the relevant file from the yarn release (I think it is the minified single file bundle, IIRC).

[telamonian]

Can we experiment without committing? I'd hate to have to delay 2.0 because we were waiting on yarn.

Those are definitely some of my reservations. Instead of the main 2.0, yarn-2.0 may deserve to be its own branch, at least at first.

[bollwyvl]

We just copy the relevant file from the yarn release (I think it is the minified single file bundle, IIRC).

Yeah, the legacy one. It's slightly bigger. But who's counting?

The yarn 2 features indeed are looking good... perhaps we'll finally be able to break free of the npmjs.org dependency. Always seemed so close with the offline-mirror stuff, but the new-fangled yarn.lock might finally be the ticket. yarn.lock.d? The pnp stuff is also really quite cool, and i guess since berry is on typescript, they must have figured the issues they were having out.

Also, for auditing purposes, it would be fantastic to be able to split the node_modules and yarn.lock of what we build with from what we ship. I'll try the yarn audit thing from #6336 again, but it crashed most of the time because of the sheer number of dependencies. For an end user (of some lab), they probably don't care if the webpack their lab was built with has a scary new vulnerabilty, but react-foo-bar-baz having one might be a big deal. This would have the down-side (?) of extensions not being able to add new loaders, or totally abuse it to ship language servers. But if that was stable during the course of a (major) release, it would save a lot of downloading, solving, rebuilding, etc.

Back on size: If we can't figure out how to have an extend-by-putting-serveable-assets-on-disk approach, I would like to see the 2.0 line perhaps break jupyterlab into more pieces, or migrate more stuff into jupyterlab_server. I've been shipping a number of labs for users of various sophistication recently, and because we always end up building and shipping another lab+extensions have been tempted to remove the assets and yarn from site-packages by patch. Feels dirty. Anyhow, jupyterlab would be "gold master" built static, and would depend on jupyterlab-core for just the serverextension, apps, and templates. I don't know where yarn would go.

We could experiment with this on conda-forge already with some outputs, and just hard-pin all of the subpackages, but it might be worth moving it more into a pip-level concern.

[jasongrout]

An update on 2.0:

I think we are all in agreement that the primary feature of JLab 2.0 is that it ships in January 2020.

A bunch of core developers met for a week in Austin at the beginning of December 2019. We decided to release a beta Monday, 23 Dec, which would be our API freeze. However, the beta released that day was broken (it could not jupyter lab build, and so you could not install extensions). Also, the release commits and tags were not pushed to master.

The build problems have been fixed in master at this point, but we still don't have the release commits or tags in GitHub.

Additionally, @telamonian has been working on finishing up a backwards-incompatible api-changing revamp of the icon system (see #7299 and #7700).

Since the first 2.0 beta was broken, I think that leaves us a window to have #7700 (if it is finished quickly) or other smaller backwards-incompatible changes merged in before the next beta.

What do people think about releasing another beta this week or early next week, and extending the api-breaking merge window until that release?

I'm hesitant to release another beta until the first beta's release commits and tags are merged into master. Note that since some PRs have since been merged into master, the release commits will need to be merged, not pushed, to master.

CC @blink1073

[jasongrout]

Also, the release commits and tags were not pushed to master.

Following up on this on gitter, FYI:

@jasongrout:

Love to hear your thoughts on #7230 (comment) - and in particular if you still have the 2.0 beta commits on your computer

@blink1073:

howdy @jasongrout, looking...
drat, I do not

To be fair, our current release process puts those commits in a temp directory which the computer can purge later automatically.

[jasongrout]

I notice we didn't make it to beta 2 yesterday (thanks for everyone who worked long and hard towards it!). I'm happy to release beta 2 later today if it's helpful. Is #7700 the main blocker for beta 2? Or is there something else we are really trying to get in?

[jasongrout]

An update: 2.0b2 is now released, and seems to work great! Thanks @blink1073 and all the other that helped with this.

[krassowski]

Just here to say that it would be very nice to allow extension developers to have at least a week from the extension migration guide publication to the 2.0 release (or maybe plan it this way for the next release).

Also, it might be nice to have a post on team-compass once the guide is available. And thank you for all your amazing work here!

[jtpio]

the extension migration guide publication to the 2.0 release

Does such guide already exist, or is it usually part of the changelog? (so we could start adding some items to it)

[afshin]

I would like to write a blog post to help users with this. It's on my agenda for this week ... hopefully.

[saulshanabrook]

@jtpio I believe we have put it as part of the changelog in the past (https://jupyterlab.readthedocs.io/en/stable/getting_started/changelog.html#extension-development-changes), so any PRs to add to it would be welcome!

[jtpio]

Thanks @saulshanabrook, that looks like a good place to put the list of changes.

[jasongrout]

I'm working on getting ipywidgets (and especially custom ipywidgets) to work in JLab 2.0, and it may be a mess for end users to understand for some third-party packages, primarily because we (a) want one widget package to support jlab 1.0, jlab 2.0, and classic notebook, and (b) some widget packages apparently have to import and use phosphor. See jupyter-widgets/ipywidgets#2781 (comment) for more details (we find there that bqplot uses phosphor, for example, to send custom messages).

My thinking is: if Phosphor is really a frozen project, is there really any harm in aliasing phosphor to lumino, at least for a deprecation period? That would allow an extension author whose api hasn't changed to make a few minor tweaks to support both jlab 1.0 and jlab 2.0 (which would be great, as it gives people 6 months to move from jlab 1.0 to 2.0 and still enjoy extension updates).

For ipywidgets, it would allow custom widget authors to be able to make the changes at a more natural ipywidgets 8 boundary in the next 6 months, rather than trying to force an upgrade for jlab 2.0, then another for ipywidgets 8.

I guess my point is that it makes it easier for people to have a deprecation period with phosphor (even just 6 months until jlab 3.0), and it doesn't really harm anything, so why not do it?

This is more of a heads-up that this abrupt switch from phosphor to lumino with no deprecation period may cause a lot of headache for extension authors in the short term. I think we could make a 2.1 (or perhaps even a 2.0.1) that enables webpack resolutions to alias phosphor to lumino during a deprecation period, so this may not need to hold up a 2.0 release.

Thoughts about re-examining the phosphor aliasing discussion in light of some of the difficulties we are seeing in supporting extensions?

[saulshanabrook]

Wait so does ipywidgets depend on bqplot which depends on phosphor?

[saulshanabrook]

I would think that ipywidgets has some core package to handle the logic shared between notebook and all lab extensions, then it also has a jlab extension that depends on that, which would have different versions for jlab v1 and for v2 (it wouldn't support both at once).

[saulshanabrook]

I would expect custom widget packages to have two versions, one for jlab 1.0 and for jlab 2.0, if they depend on phosphor. If they were also jlab extensions themselves, then that should be fine as well? Is the issue that in ipywidgets output you hard code a link to a custom widget package JS library and use require to fetch it?

[jasongrout]

I would think that ipywidgets has some core package to handle the logic shared between notebook and all lab extensions, then it also has a jlab extension that depends on that, which would have different versions for jlab v1 and for v2 (it wouldn't support both at once).

Yes, that's basically how it works. The key here is that bqplot on top of all of this has one codebase, but it imports phosphor to work with the underlying phosphor widgets.

Fortunately, because we have the basic structure you outline above, most widget packages I think will work just fine in jlab 1.0 or jlab 2.0. There are some corner cases, though, that are made sharper and more inconvient for everyone because of our abrupt no-deprecation change from phoshor to lumino.

[jasongrout]

There are some corner cases, though, that are made sharper and more inconvient for everyone because of our abrupt no-deprecation change from phoshor to lumino.

My basic point is that we have a choice: we could take the position that we want to support extension developers as much as we can, so we'll enable a deprecation period to give them time to move over if the phosphor/lumino change is the only thing affecting them, or we could take the hard-line approach that we are moving over now and everybody has to move with en-mass with deprecation period. I think while we are trying to gain the trust of the extension authors, and while we are trying to encourage a robust ecosystem to grow, it is prudent to go with the first approach.

[saulshanabrook]

I agree with you about the choice, but I don't think it stops with the phosphor -> lumino change. It seems just as applicable for any backward incompatible change from jlab 1.0 to 2.0.

If we want to help extension authors not have to create a new version that targets 2.0, then I think we just shouldn't have a major version bump of the packages because the semver means "things might break".

So it seems like maybe its a bit too late to do that? Unless we wanna re-work all our packages to be backwards compatible with the 1.0 specifications?

[saulshanabrook]

(for some context, I am a bit hesitant to start doing aliasing because it's not checked by typescript and introduces even more complexity and uncertainty into our build system).

[saulshanabrook]

@jasongrout if you wanna hop on zoom quickly, we could chat about this in real time then document our notes here.

[jasongrout]

@jasongrout if you wanna hop on zoom quickly, we could chat about this in real time then document our notes here.

Sorry, running on some tight timelines today.

Here's the scenario I'm trying to help:

Sally just developed an extension in November for JupyterLab 1.0. Because some of her stakeholders can't upgrade until the end of the academic year, she must support the jlab version 1 of her extension until, say, JupyterLab 3 is released, but she also wants to release a jlab 2-compatible version. Now, it turns out that there were no breaking changes in JupyterLab API that affect her extension, so she actually could change her requirements for the jlab packages to be the jlab 1 dependencies OR the jlab 2 dependencies. It could even just be a minor release of her extension! This is really nice, since it means she can maintain a single codebase for JLab 1 and JLab 2. However, she realizes that since she used a phosphor package (it seemed encouraged in jlab 1, right? For example, she is making a new widget to add to the dock area), she actually can't do this. It seems a bit unnecessary to her, since she understood that the last phosphor release and the first Lumino release were exactly the same, and the idea was to support people moving from phosphor to Lumino transparently.

We could let her support both JLab 1 and JLab 2 with the same codebase. It seems unnecessarily harsh to force the migration when a couple of lines in our configuration could solve the issue for her and make her life much easier.

To use another analogy, we could force people to rip the bandaid off, or we could, with a couple of configuration lines and (probably?) no ill side effects, we could give them 6 months to slowly take the bandaid off, if that's the way they want to do it.

I'm only advocating for a deprecation period for the phosphor->Lumino transition, because that's very easily solved on our side (I think), and I think it sends a good signal to extension developers that we have your back.

[jasongrout]

I agree with you about the choice, but I don't think it stops with the phosphor -> lumino change. It seems just as applicable for any backward incompatible change from jlab 1.0 to 2.0.

I think the phosphor -> lumino change is a particularly good change to deprecate, since it is (a) a straight rename, no backwards-incompatible api changes, (b) we explicitly were trying to make the migration a good experience, and (c) it seems like it will be pretty trivial on our side to support.

That said, I won't be able to work on it before next Wednesday, so if we proceed with the release, fine - we can look at putting this deprecation back in in 2.0.1, I think?

[vidartf]

Note that a lot of third-party ipywidget extensions import phosphor typings, even though there are no direct imports of run-time objects. E.g. for one widget library I have, updating to the latest ipywidgets RC (npm install --save @jupyter-widgets/base@next @jupyter-widgets/controls@next), gives these Typescript compile errors for the lab plugin (the only part that fails):

> tsc

src/plugin.ts:31:3 - error TS2322: Type 'Token<IJupyterWidgetRegistry>[]' is not assignable to type 'Token<any>[]'.
  Type 'Token<IJupyterWidgetRegistry>' is not assignable to type 'Token<any>'.
    Types have separate declarations of a private property '_tokenStructuralPropertyT'.

31   requires: [IJupyterWidgetRegistry],
     ~~~~~~~~

  node_modules/@phosphor/application/lib/index.d.ts:44:5
    44     requires?: Token<any>[];
           ~~~~~~~~
    The expected type comes from property 'requires' which is declared here on type 'IPlugin<Application<Widget>, void>'


Found 2 errors.

[vidartf]

(to be explicit about the previous comment: In such cases, the widget library authors will need to cast to any to enable support for both lab 1.x and 2.x in one release, AFAICT)

[jasongrout]

I updated the todo items in the OP.

[jasongrout]

Congratulations and thanks everyone!

[jasongrout]

We will be shipping 2.0.2 next week, after which we can close the 2.0 milestone.