Ship vega4-extension by default

[davidanthoff]

I brought this up at the end of #6402, but given that this is really a separate action item from that original issue, I'm opening a new issue here.

I think jupyterlab 1.0.0 should ship the three extensions I mentioned in the title out-of-the box, in addition to the vega5-extension that is already included by default. I think ideally this issue here should be added to the 1.0.0 milestone.

Without that, users will experience a major regression when they update to jupyterlab 1.0.0: all notebooks that they created with earlier versions of jupyterlab that include vega or vega-lite figures will no longer display properly when opened in jupyterlab 1.0.0. I feel that experience would be incompatible with the promise that was made last spring that juptyerlab was ready for end users for their daily work. That messaging implied (to me at least) that minimally I would be able to open files in jupyterlab 1.0.0 that I created with vanilla juptyerlab 0.35.x and they would display correctly.

I believe this affects minimally Python and Julia users: Python users that used altair in their notebooks, and Julia users that used VegaLite.jl. I don't have numbers, but it seems to me that both packages were quite popular over the last year, so this doesn't strike me as a corner case/small user group that would be affected.

This would also be in line with what nteract is doing with their support for vega and vega-lite out-of-the-box.

I don't know how complicated this would be, and I do understand that it is late in the release process for jupyterlab 1.0.0... But on the other hand, I think from an end-user point of view this is really a major regression/bug. But maybe this wouldn't be so complicated to fix? Might it be enough to just move the existing code for vega2-extension, vega3-extension and vega4-extension from https://github.com/jupyterlab/jupyter-renderers into this repository here? And add them to a few config places so that they get shipped out-of-the-box?

Pinging @domoritz and @jakevdp from the vega, vega-lite and altair side of things. Not sure who from the jupyterlab team would be the right point person?

[jakevdp]

Thanks for opening this, @davidanthoff. I think this is a good suggestion.

[domoritz]

I'm supportive of this. Since Jupyterlab loads Vega asynchronously the bundle shouldn't be larger either. I don't think we want to go back to ancient Vega and Vega-Lite versions, though.

[jasongrout]

I don't think we want to go back to ancient Vega and Vega-Lite versions, though.

What is 'ancient' in your mind?

[jasongrout]

Tagging as 1.0 for discussion.

[domoritz]

I don't think anyone needs versions before Vega-Lite 2 and Vega 3 in JupyterLab.

I would say we concentrate on what Altair needs and that is Vega-Lite 2 and Vega-Lite 3. The corresponding Vega extensions in jupyterlab are vega4 and vega5.

[domoritz]

My suggestion would be to bring back the vega4 extension from before #6133. Any objections? If not, I will send a PR.

[jasongrout]

So to confirm, we'd just be adding one more extension, the vega4 extension? Would you also update it to pull in the most recent appopriate vega-embed: https://github.com/jupyterlab/jupyter-renderers/blob/a3a5d361eefbe1a8ab5519e7658f7e5bbb74af91/packages/vega4-extension/package.json#L35

[davidanthoff]

My suggestion would be to bring back the vega4 extension

I think that makes sense for another reason: Jupyterlab v0.33.0 was the first release that was not billed as a beta release and declared as ready for general use. That version had vega4 in it, so I think supporting that version that was declared as ready for general use going forward as the oldest release is reasonable.

[jasongrout]

Changing title to reflect this.

[ellisonbg]

A few of us are talking about this at the JLab 1.0 sprint. Questions that are coming up:

1. What are the downsides, especially in terms of performance or UX of shipping both vega4 and vega5?
2. If we ship vega4 and vega5 by default now, what happens when vega6 comes out?
3. Did we move to ship vega5 by default too quickly? In other words, should we only ship vega4?
4. Is all of this simpler if we don't ship any of them by default?

Of these, getting clarity on 1+2 is especially important to move forward with shipping both vega4 and vega5. Thoughts?

[domoritz]

1. As far as I understand it, there are no downsides. The majority of the extension is loaded asynchronously so the impact is minimal. For users the UX will be better since they can use Vega 4 and 5 at the same time.
2. We ship vega6, I guess.
3. Then Altair 3 wouldn't work. I think this would be the worst possible situation for users.
4. I don't think so. JL needs to have a story for updating versions of dependencies and Vega is probably a good test case because you have me and I can make releases and fix incompatibilities for you ;-)

Another option may be a Vega package that supports multiple versions at once but I won't have the cycles to implement this.

[jakevdp]

I wonder if it might be possible to remove the versioning entirely from the vega extension. Can lab extensions access HTTP as part of their handling of different mimetypes? If so could there be a versionless vega etension that uses the vega version listed in the the specification's $schema argument, loaded via the standard CDN links? The downside would be that it would no longer work offline. The upside would be that it would just work and no longer depend on your kernel's Python packages being in sync with your kernel manager's frontend extensions, and that would be a HUGE win as far as usability and maintainability.

[domoritz]

Or the extension could come with one version that works offline and for older versions it fetches via CDN. I like that.

The tricky bit is that Altair sets the mimetype to be a particular version of Vega-Lite rather than just Vega-Lite. This means that the extension won't be loaded. Would it make sense to not output a versioned mimetype from Altair?

[jakevdp]

It would be easy to change Altair to output an unversioned mimetype, if the frontends would support it.

[davidanthoff]

I think having one extension that handles all vega and vega-lite versions would be great. Moving to one version independent MIME type also sounds like a great idea. I think ideally such an extension would a) ship offline versions of all major versions of vega and vega-lite that existed when the extension was last shipped out-of-the-box, and b) had an option to download (and cache for offline use) newer versions if it encounters a spec that had a newer version header.

For 1.0.0, though, I think it would be great if the priority was to ship something that a) displays all existing notebooks that were created with released versions of jupyterlab out of the box (so that means it has to support versioned MIME types that were in use over the last year or so) and b) includes support for vega-lite 3 and vega 5 (which I think means just adding the existing vega4 extension to jupyterlab). It seems that just moving the vega4 extension is the more realistic option in the short term that would also not derail the schedule for jupyterlab 1.0.0 too much?

[ellisonbg]

Thanks for giving input on this everyone. I don't think relying on a CDN for extensions that are shipped with JupyterLab is going to work. We have a lot of people who run JupyterLab in locked down settings where they aren't going to be able to reach a CDN.

I agree it would be nice to have a single, unversioned mime-type for vega and vega-lite, but given how close we are to 1.0, that will probably have to wait until later. This work will also impact lots of projects (nteract, colab, altair, etc), so it will take some careful planning.

Based on this, my sense is:

• For the short term, ship the vega4 and vega5 pacakges in JupyterLab 1.0.
• Long term, create a new unversioned mimetype, a single renderer for that, and a version of vega-embed that can render those things.

Is everyone ok with this plan?

[jasongrout]

@domoritz - to test this out, can you give us a vega 4 file that will not work in vega 5, and a vega 5 file that will not work in vega 4? Then we can be sure we are using the right versions for each file.

[jasongrout]

@davidanthoff - the notebooks you posted about over at nteract/nteract#4442 containing vega 4 and vega 5 code - are they invalid for the other version of vega? Just curious.

[davidanthoff]

No, I think they are probably exactly the same JSON, just stored as a different MIME type.

[saulshanabrook]

@jasongrout You can do this by installing altair<3 and altair>3 and running a simple example like:

import altair as alt
import pandas as pd

source = pd.DataFrame({
    'a': ['A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I'],
    'b': [28, 55, 43, 91, 81, 53, 19, 87, 52]
})

alt.Chart(source).mark_bar().encode(
    x='a',
    y='b'
)

[jakevdp]

No need to install multiple altair versions. You can do something like this with Altair v3:

# Vega-Lite 3 chart. This spec would fail if passed to vega-lite 2 because of the errorbar mark
import altair.vegalite.v3 as alt
from vega_datasets import data

alt.Chart(data.barley.url).mark_errorbar(extent='ci').encode(
  x='yield:Q',
  y='variety:N'
)

and:

# vega-lite 2 chart. This would render under vega-lite 3 because of backward compatibilitiy
import altair.vegalite.v2 as alt
from vega_datasets import data

alt.Chart(data.barley.url).mark_rule().encode(
    x='ci0(yield):Q',
    x2='ci1(yield):Q',
    y='variety:N'
)

I'm having trouble coming up with a spec that would be valid in Vega-Lite 2 but invalid in Vega-Lite 3... the team did a pretty good job of ensuring backward compatibility.

[jasongrout]

If we can't find a chart that won't render with vg 3, then perhaps this discussion is moot?

[jakevdp]

I don't think it's moot: we still need a v2 entrypoint, which is provided by the Jupyter extension, or charts created in old versions of Altair will not render. And my knowledge of the schema is not exhaustive, so I can't confidently say there are no examples that render in V2 and not in V3; I just can't come up with one.

[domoritz]

I'm having trouble coming up with a spec that would be valid in Vega-Lite 2 but invalid in Vega-Lite 3... the team did a pretty good job of ensuring backward compatibility.

You're welcome :-)

I think the only backwards incompatible change is that maxTitleLength is now titleLimit. See https://github.com/vega/vega-lite/releases/tag/v3.0.0

[jakevdp]

I think the only backwards incompatible change is that maxTitleLength is now titleLimit

Yeah, I spent some time playing with that, but I haven't figured out how to create a chart where titleMaxLength has any visible effect in VL 2. The docs say it's "Max length for axis title if the title is automatically generated from the field’s description," but I'm not certain what that means (I tried using an aggregation, so that the title would be generated from the aggregation and the field name, but the parameter still has no apparent effect).

And in VL3 it doesn't result in an error, even though it's not part of the schema.

[domoritz]

I'm getting an error

[jakevdp]

Yeah, there's an error in the vega editor's linter, but not in an embedded chart.

[jakevdp]

Is there a way to make that error show up within the JupyterLab extension? That's what I can't figure out how to do.

[jakevdp]

I'm trying to provide what @jasongrout asked for in #6572 (comment)

[domoritz]

Oh, the editor used the schema to validate a spec and shows warnings. Vega-Embed doesn't validate inputs.

[domoritz]

Vega embed will check the $schema and compare the version in it with vl.version or vega.version. I already talked with @saulshanabrook about confirming that the versions are right on Slack.

[jakevdp]

Oh, OK 👍

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related discussion.