Support control<>iopub messages to e.g. unblock comm_msg from command execution

[tkrabel-db]

Disclaimer: I work for Databricks, where we patched the control channel to handle jupyter messages like comm_open and comm_msg.

What changes with this PR?

At Databricks (which uses ipykernel), we often face the situation that customers execute long running pyspark jobs which block the shell channel for minutes or even an hour. In order to provide a better user experience, we unblock some original shell messages by moving them over to the control channel. We successfully did that with complete_request, comm_open and comm_msg.

Part of our fix was changing the comm object so that it gets the parent header from the correct thread. My PR is about that change.

[blink1073]

Thank you!

[tkrabel-db]

@jasongrout FYI. This went faster than expected :)

[maartenbreddels]

Just for the record: this does not change existing behavior? This only supports your use case of sending back to the right channel if that happens. So this is totally backward compatible, right?

[tkrabel-db]

Yes. The existing behaviour is shell<>iopub channel communication, which still works as expected. This change supports adding control<>iopub channel communication without the necessity of patching internal ipykernel functions, which is what we had to do at Databricks so far.

[jasongrout]

Awesome, thanks for reviewing this, @blink1073.

I talked with @minrk about this yesterday night too, and he suggested that this if statement would be better positioned in the get_parent function, so that it could be used by not only comms, but by anyone getting the appropriate parent header in a thread. More concretely, he suggested the logic for picking the right parent header be implemented at

ipykernel/ipykernel/kernelbase.py

Line 632 in d63c33a

return self._parents.get(channel, {})

and we change the signature of get_parent to something like get_parent(self, channel=None) (where the default None value means to use this logic to determine the correct parent header).

@tkrabel-db - what do you think?