Notify the user when a notebook kernel autorestarts

[jasongrout]

References

Fixes #4273

This tackles some of the same issues as #4724, but is way less ambitious.

Code changes

• Restarting a kernel now does not disconnect the websocket, but keeps the same websocket open. This helps us get all the messages from a restarted kernel. The logic around the kernel ready promise and status was reworked to reflect basically connection status (i.e., a kernel is not ready if it is not connected, but messages will still be queued and sent when ready automatically).
• The kernel state transition logic is refined
• A new autorestarting kernel state is added for when a kernel is restarted automatically by the server (for example, if the kernel runs out of memory and dies, the server recognizes this from the heartbeat, and tries restarting it automatically)
• A dialog is popped up the first time an autorestarting status is set in a row in a notebook, notifying the user that a kernel is being restarted automatically. Like the classic notebook, if multiple autorestarts happen in a row, only one dialog pops up.
• Documentation for shutting down a kernel that is already dead is changed to match existing behavior (i.e., shutting down a dead kernel does not return a rejected promise, but just clears the websocket and returns normally).
• jlpm run coverage now does not bail on the first failed test, but instead runs through all tests. This is so we can see all test failures instead of just the first one in CI.
• Converted more promise-based code to async/await
• Kernel busy status indicator now shows busy when state is busy, starting, or restarting, instead of just showing idle when state was idle. Perhaps state should also show busy when state is autorestarting?
• Consoles now don't make a separate kernel info request to get the banner, but instead just wait until the kernel is ready and use the cached kernel info.
• Kernel messages are saved from the very start, and sent as soon as the kernel is ready (which means as soon as a request kernel info request succeeds, so the kernel is a known idle state). This means you don't need to wait until a kernel is ready before sending a message, and you can just await the resulting future.done promise. This cuts down on a lot of test console errors where the kernel was disposed before it was done getting replies back.
• The logic around disposing kernel futures on kernel disposal was reworked to be more careful about edge cases.
• Several tests are restructured to cut down on the number of kernel startups in tests to save time.
• Fixed a few other errors in tests, and now run a few tests that had been skipped because they spuriously seemed to fail (around kernel restarts)

User-facing changes

A dialog pops up in a notebook when a kernel is autorestarted, with the same text as the classic notebook.

import os
os.kill(os.getpid(), 9)

Backwards-incompatible changes

• Kernel state transitions are subtly different and have a new state, autorestarting.

[jupyterlab-dev-mode]

Thanks for making a pull request to JupyterLab!

To try out this branch on binder, follow this link:

[jasongrout]

One of the things I am seeing in my logging is that if I open a notebook, then open an associated console, then close the associate console, the websocket for the console kernel connection stays open, which means that it receives all of the broadcast iopub messages. We should probably close the kernel connection when a console is closed.

Edit: I opened #6290 to deal with this issue

[jasongrout]

This is now getting much closer. Restarts now don't disconnect and reconnect the websocket. The kernel status should switch from restarting to autorestarting when an autorestart happened, so we can listen for that state to notify the user. Popping up a dialog is the next step.

[jasongrout]

@blink1073, @ian-r-rose, and everyone else - I think this is ready for review.

[blink1073]

Thanks for digging through that state machine, nicely done!