No warning/error message when kernal is forced to restart

[NicWayand]

Sometimes when running a notebook the kernal restarts (in my case due to a OOM issue) but does not give the user any message. The cell will still show an asterisk, as if its running, but only when the user hovers the mouse over the circle at the top right, will it show that the kernal is restarting.

I would expect a larger warning to the user, either in the current cell, or at the top of the notebook, to let them now some issue has occurred.

In the terminal where jupyter lab was started gives:
KernelRestarter: restarting kernel (1/5), keep random ports

jupyterlab 0.31.12

[clkao]

This happens in 0.34.7 as well.

[clkao]

Note for plain old notebook, there's a simple popup for OOM/restarted kernel, which should be informative enough.

[kykrueger]

I'd like to see more details as to why the kernel has had to restart. Some sort of stack trace from what the kernel was trying to execute.

[jasongrout]

That would be great, however if it was an OOM, it's killed suddenly without a stack trace, IIRC.

[kykrueger]

How can we tell if it was an OOM problem?

[jasongrout]

With our current infrastructure, I'm not sure you could easily. The process is gone and we just notice it no longer is responding to heartbeats.

[kykrueger]

Gotcha,

Some of the jupyter docs for the architecture were really helpful for understanding where the limitation lies. Would I be correct to assume that this crash and restart is caused by a timeout between the kernel and notebook server?

Is this something that the notebook server has an interface for, but the kernel has not implemented?

I'll try and get some errors out of my installed version of ipykernel later.

[jasongrout]

Is this something that the notebook server has an interface for, but the kernel has not implemented?

Right. There is a message the server sends the frontend that indicates the restart happened without a user request, and JupyterLab currently is not distinguishing between that and a restart requested by the user. The classic notebook notifies the user if a restart happens that was not requested.

I worked on this and many other things in #4724, but that turned into a much more ambitious project that should be postponed until after 1.0. I still plan to look again at this specific issue to investigate solving just it (#4748 is the 1.0 issue for this, though perhaps this should be the 1.0 issue and that should be closed as a duplicate).

[jasongrout]

I still plan to look again at this specific issue to investigate solving just it

And as with any issues, help is welcome! If anyone wants to take this up, please comment on the issue so that work isn't duplicated.

[kykrueger]

I worked on this and many other things in #4724, but that turned into a much more ambitious project that should be postponed until after 1.0.

@jasongrout that looks like quite the rabbit-hole. I'd be quite happy if we can solve this separately. I've never used TypeScript or JavaScript, but I'll see if I can help.

[jasongrout]

For reference, here is where that message is displayed in the classic notebook: https://github.com/jupyter/notebook/blob/eea37a6b2110e0b8a5491f17e241fa3c8aecb5f7/notebook/static/notebook/js/notificationarea.js#L122-L146

and it is triggered here: https://github.com/jupyter/notebook/blob/eea37a6b2110e0b8a5491f17e241fa3c8aecb5f7/notebook/static/services/kernels/kernel.js#L1106-L1113

[jasongrout]

It seems that killing the kernel process so that it autostarts does not work in jlab (I can't use the newly started kernel)