[do not merge] Naive JS Comms

[bollwyvl]

So this... kinda re-implements ipykernel's CommManager, and slaps it on all the kernels, plus adds some widgets.

I am still getting my feet under me as to the local approaches vs previous endeavors, and will admit there was even more as any going on, but it's kinda into shape, from a soundness perspective 😊.

References

• a potential approach to (part of) Comms support #18

Code changes

• adds a DefaultCommManager to all kernels
  • can be overridden by subclasses
• adds a barebones JS widget implementation
  • this approach is not going to scale, but need to show a few things!
    • HasTraits
    • Widget
    • FloatSlider
    • widget.observe
    • link
    • dlink
    • jslink
    • jsdlink
    • Textarea
    • Select
    • HTML
    • Upload (for buffers?)
    • interact?
• in the JS Kernel, adds self.kernel, and widgets, pretty much defeating the iframe isolation 😱

User-facing changes

Users of the docs site would be able to use at least a couple FloatSliders in JS 😉 :

I did originally dogfood the whole thing in a JupyterLite JS notebook... it was kind of hair-raising, and we gotta work on our persistence game.

Backwards-incompatible changes

• Haven't thought about it much... if someone already wrote a comm implementation and is holding it back, it would no doubt be incompatible 😊

Future Work

• do a data-driven lift of the upstream as a file (e.g. JSON Schema) and generate classes
  • types are fun and all, and probably table stakes for testing,
    • but we want concrete (generated) js implementations that will work with (eventual) completion in bare JS
• proxy this api again back into daughter-of-pyolite Rename the Pyolite kernel #116 across the WebWorker divide
  • create ipykernel, ipython, ipywidget shims
• make the CommManager more pluggable
  • alternate offer contracts than OneCommManagerPerKernel
    • localForage
    • Yjs-backed

[bollwyvl]

Gross:

/home/docs/checkouts/readthedocs.org/user_builds/jupyterlite/checkouts/141/docs/api/ts/interfaces/kernel_src.icommmanager.itarget.md:10: WARNING: Non-consecutive header level increase; 2 to 4 [myst.header]

[jtpio]

Really neat, thanks for starting this!

[jtpio]

Playing around on the RTD preview:

js-widgets.mp4

[martinRenou]

You're amazing!

[martinRenou]

@jtpio can you share a link to the RTD preview?

[jtpio]

can you share a link to the RTD preview?

It's in the checks below:

[jtpio]

This one should be fixed on main now.

[martinRenou]

We should find a way to plug your comm implementation to ipywidgets. Do you think monkey patching ipykernel could be enough? something like import sys; sys.modules['ipykernel.comm'] = your_comm_module should do the trick, this is what we do in xeus-python and it works like a charm. I would agree monkey patching is an ugly thing to do. But unless ipywidgets is changed, there was no way to do otherwise.

The only problem is ipywidgets bringing ipykernel, we should install ipywidgets ignoring dependencies. I don't know if conda can do that.

[jtpio]

Do you think monkey patching ipykernel could be enough?

Maybe a shim will be enough? That would mean creating a pure python package for ipykernel that would be installed in the pyodide kernel by default. And probably the same for ipython. We could also add first-party support for ipywidgets so users don't need to micropip.install('ipywidgets').

[bollwyvl]

Yeah, I dunno just yet about how to make this quack right in pyolite with actual packages... it'll be interesting, of course, but maybe still outside the scope of this PR.

[jtpio]

Wondering whether we should adopt the same pattern as for the other messages, by having methods on the IKernel interface?

https://github.com/jtpio/jupyterlite/blob/97bbe1a0e8ec31108d905c3b7752666a01634851/packages/kernel/src/tokens.ts#L45

For example with commOpen, commMsg and commClose.

This could allow other kernels to completely change the implementation of the comm manager if they want, or just forward messages in the case of the Python kernel reusing the existing comm implementation.

[bollwyvl]

Yeah, as i mentioned, i'm still figuring out the architecture. the joke was, of course, first try to sneak into how one might get this into pyolite, realizing that was way crazy, and then unwrapping the actual flow of messages through the system to get to something that works.

This would of course need to move into its own package, but no reason not to have the API in better shape...

[martinRenou]

In the pyolite PR https://github.com/jtpio/jupyterlite/pull/145, the approach I took was to more-or-less copy/paste the ipykernel comm implementation (which we could make a separate package so that both ipykernel and pyolite could use it).

This means the comm implementation is in Python, and lower in the architecture. So the approach to make those methods abstract as Jeremy is suggesting makes sense.
Now I am wondering if it's feasible and if it makes sense to reuse @bollwyvl 's JS implementation of the comms instead, we would need to expose those comms to the "Python process".

[jtpio]

Now I am wondering if it's feasible and if it makes sense to reuse @bollwyvl 's JS implementation of the comms instead, we would need to expose those comms to the "Python process".

In theory probably yes (or hopefully yes), if the JS implementations follow what ipykernel expects. This could be passed as a custom Python package from JavaScript: https://pyodide.org/en/stable/usage/faq.html#how-do-i-create-custom-python-packages-from-javascript

[bollwyvl]

Right, I think we win if these two PRs can be harmonized, but prettied up, moved to some new packages....

packages/
  kernel/
    kernel.ts        # gets the comm(Open|Msg|Close)Msg handlers
  js-kernel/
    kernel.ts        # has new CommManager({kernel: this})
  py-kernel/src/
    worker.ts        # also needs CommManager in here...
    kernel.ts        # ... injects it here e.g. `import js; (await js.import(${COMM_URL})).CommManager`
    py/.../
      patches.py     # sys.modules hacks, as on #145
                     # ...and the shim to do-async everything from #141 js
  comms/src/
    comm.ts
    comm-manager.ts
    providers/
      memory.ts
      # TODO: localForage.ts
      # TODO: yjs.ts

In patches.py, there are probably some more pointy bits... At the pyolite/import js boundary we need it to pretty much be dumb JSON types and Arrays, but it should be feasible.

[bollwyvl]

I'll hoist this back to #18