Remove vega 4 extension

[jasongrout]

In JupyterLab 2.0, I think it makes sense to finally remove the vega 4 extension and move to vega-lite 4.0 (which should support vega-lite 3 files). See #7428 for many more details and commits about this upgrade.

[domoritz]

Vega-Lite 4 is out now. We have not sent the announcement yet but it's on npm.

[davidanthoff]

So I'm quite nervous about this, after reading through all of #7428.

I think one of the core scenarios for JupyterLab notebooks is replication code for scientific papers. In those scenarios, you publish a paper, store a copy of your replication code on a service like zenodo.org, and then folks will try to look at and use that notebook for years to come. I think literally we need to think 5-10 years here, minimally. The cycles in science are just very, very different than in a more normal software environment. I think one of the core promises of Jupyter notebooks (if it wants to target the scientific space) must be that you can still open notebooks from way back in a recent version of JupyterLab, and everything will display just fine and in the same way it did when the notebook was created. Ideally you could also run it, but that is more complicated. But displaying seems like the minimal bar to me.

So these two quotes from @domoritz in the thread in #7428 make me nervous:

I think always keeping the last two versions would be good for users.

In rare cases, we don't do this for example in Vega-Lite 4 where we are removing support for rangeStep. I don't think we can promise backward compatibility for all future versions but we are making an effort to stay compatible.

I just honestly think that doesn't jive with the needs of replication code in the scientific domain...

Wasn't the original plan that JupyterLab would just ship with multiple (major) versions of vega and vega-lite, and would use the appropriate version to render cells in notebooks, based on the vega(-lite) version encoded in the MIME type of the cells? And that old versions wouldn't be removed, but just carried along? That seems to be the most stable approach to guarantee that old notebooks are rendered the same way they were when they were created? In my mind we should make every effort to stick with that plan, it seems like the right approach to me. If the problem was that the older versions of vega and vega-lite were incompatible with newer TS, then maybe we should try to release patch releases of the older versions of vega and vega-lite that work with the newer TS.

There is another story here: the Julia VegaLite.jl package always embeds a vega-lite MIME representation and a PNG representation in notebooks. So we always have one MIME representation that will look exactly correct in every notebook. What would be an unfortunate scenario is one where say JupyterLab 5 used vega-lite 6 to render vega-lite 3 MIME types, but due to some corner incompatible change it rendered it incorrectly. I would almost prefer if in that case the vega-lite 6 renderer just declared that it can't render vega-lite 3 MIME types, and JupyterLab fell back to just showing the PNG version. But really I think what should happen in that case is that JupyterLab 5 should still ship vega-lite 3 and use that to render the plot :)

I think the question of how to show vega and vegalite files is easier, I think just using the latest version of vega(-lite) seems ok for that scenario to me, if there is no version schema embedded in the file itself.

[davidanthoff]

Having said all of that, from my point a priority would be to get vega-lite 4 support into the shipping product soon, and if that means that for a while support for previous versions is not ideal, that would be ok with me, as long as we could find a medium run strategy to include first-class support for old versions of vega and vega-lite :)

[domoritz]

In rare cases, we don't do this for example in Vega-Lite 4 where we are removing support for rangeStep. I don't think we can promise backward compatibility for all future versions but we are making an effort to stay compatible.

Just a bit more context. We remove rangeStep from the schema but the compiler translates it automatically to the equivalent properties in Vega-Lite 4. Backwards compatibility is important to us (because of reproducibility as you mentioned). We will always make an effort to maintain backwards compatibility. However, we won't be able to guarantee it if there is a conflict with a new feature. When we break backwards compatibility (for some reason), we will throw a warning/error. In #7428, we were discussing different options and not necessarily a plan.

I would almost prefer if in that case the vega-lite 6 renderer just declared that it can't render vega-lite 3 MIME types, and JupyterLab fell back to just showing the PNG version

I think that's the plan. Only if Vega-Lite is fully backwards compatible, we will accept it.

I hope that my explanations give some context on the commitment to backwards compatibility from the Vega-Lite team. I'm also very happy to discuss better strategies on what we can do from the Vega-Lite side. Having said that, I don't have any say in the versioning strategies for jlab and refer to @jasongrout for comments on that side of things.

[jasongrout]

Wasn't the original plan that JupyterLab would just ship with multiple (major) versions of vega and vega-lite, and would use the appropriate version to render cells in notebooks, based on the vega(-lite) version encoded in the MIME type of the cells? And that old versions wouldn't be removed, but just carried along?

I never thought we'd ship old versions of vega-lite indefinitely with core. Rather, we'd ship older versions as extensions, and as a one-time exception for jlab 1.0, we shipped vega 4. Anyways, it is a moot point since the old version of vega-lite does not support new Typescript versions, so we can't include it in core.

If the problem was that the older versions of vega and vega-lite were incompatible with newer TS, then maybe we should try to release patch releases of the older versions of vega and vega-lite that work with the newer TS.

That is out of our (JLab's) hands. The Vega project has decided not to ship patch releases of the older versions of vega-lite to support the new Typescript versions.

We (core JLab) are providing whatever backwards compatibility and reproducibility guarantees that vega-lite is itself providing in its latest version. @domoritz and I also discussed how Vega-lite can provide an easy way to upgrade format versions that we can use from JLab. I think that's the most practical position for us to take for core jlab.

There is another story here: the Julia VegaLite.jl package always embeds a vega-lite MIME representation and a PNG representation in notebooks.

Agreed, I think we should do this as our fallback for backwards compatibility (i.e., encode the chart in a format like png that is widely used and has a much, much longer lifespan than the vega format).

[jasongrout]

I think one of the core scenarios for JupyterLab notebooks is replication code for scientific papers. In those scenarios, you publish a paper, store a copy of your replication code on a service like zenodo.org, and then folks will try to look at and use that notebook for years to come. I think literally we need to think 5-10 years here, minimally. The cycles in science are just very, very different than in a more normal software environment. I think one of the core promises of Jupyter notebooks (if it wants to target the scientific space) must be that you can still open notebooks from way back in a recent version of JupyterLab, and everything will display just fine and in the same way it did when the notebook was created. Ideally you could also run it, but that is more complicated. But displaying seems like the minimal bar to me.

I strongly support the ideals above. If you want to support that sort of lifespan, you have to use (or at least convert to) formats that have that sort of lifespan, and right now my understanding from the vega team is that they do not plan on supporting the vega file format on that lifespan. That means use archival static png, jpg, svg, etc. for the plots. That's why I think the strong story here for backwards compatibility is in mirroring vega plots as a static png.

[davidanthoff]

It seems that nteract is shipping all previous versions of vega and vega-lite out-of-the box, via the https://github.com/nteract/any-vega package. Could jlab maybe utilize that package as well? Presumably if you were to just use that package, then the TS issue would be moot because that package is just pure JS?

But I should also say that from my point of view it would be higher priority that vega-lite 4 is in jlab 2 when it is released than any of this other stuff.

[jasongrout]

Could jlab maybe utilize that package as well? Presumably if you were to just use that package, then the TS issue would be moot because that package is just pure JS?

Interesting!

But I should also say that from my point of view it would be higher priority that vega-lite 4 is in jlab 2 when it is released than any of this other stuff.

+1 definitely to shipping vega-lite 4.