Out-of-order execution #9566
Comments
|
Does it happen if using "run all" instead? I think the execution queue is populated differently, but still worth checking... |
|
BTW - for me it's been impossible to reproduce it even as reported above, but that can be variations in the binder node I happened to get. My kernel restarts were super fast, and execution (whether manual or "run all") came very quickly and always in-order. So it may be a bug that's a little tricky to trigger/replicate. |
No, it happens only when executing the cells manually. |
|
The browser output shows a bunch of |
|
Sounds possible, though I don't know that code at all, unfortunately... Worth perhaps instrumenting that particular path to add more info about what kind of message it was trying to send. This is definitely a pretty serious bug, as it will lead to potentially very hard to understand behavior if a user doesn't realize it happened (and depending on their code, it could still run, just produce odd results). |
Looks like the error might come from this? |
|
If it's only possible to reproduce depending on location (I opened the binder from France), I'd be happy to help with debugging. |
So it looks like it may be tied to comm messages? So perhaps widgets are a good way to reproduce it? |
|
I am not sure if this is related but I have opened another issue in which I could see |
|
I think the root problem here isn't with comm messages. I can occasionally reproduce the issue on a local install of all current packages in a fresh conda env with a notebook with simple inputs. Here is one result of restarting the kernel and immediately evaluating the cells in this notebook from top to bottom:
Conda packages: |
|
Update: I think I found a place in the logic where some messages get queued, then others are sent (jumping the queue) before the queued messages are sent. I'll investigate more later today. Relevant logic seems to be at jupyterlab/packages/services/src/kernel/default.ts Lines 385 to 395 in d5a6043 |
Maybe that explains the example above? With:
|
|
Yes, that's my hypothesis. And probably the queued messages are scheduled for sending, but async, so the direct sending cuts in line? We have to be careful, because we do want kernel info requests to jump the queue. |
|
Relevant issues:
|
Fixes jupyterlab#9566 Followup on jupyterlab#8562 Changes solution in jupyterlab#9484 If we restarted a kernel, then quickly evaluated a lot of cells, we were often seeing the cells evaluated out of order. This came because the initial evaluations would be queued (because we had the kernel restarting sentinel in place), but later evaluations would happen synchronously, even if there were still messages queued. The logic is now changed to (almost) always queue a message if there are already queued messages waiting to be sent to preserve the message order. One exception to this is the kernel info request when we are restarting. We redo the logic in jupyterlab#9484 to encode the exception in the _sendMessage function (rather than hacking around the conditions for queueing a message). This brings the exception closer to the logic it is working around, so it is a bit cleaner. Also, we realize that the sendMessage `queue` parameter is really signifying when we are sending pending messages. As such, we always try to send those messages if we can. Finally, we saw that there was a race condition between sending messages after a restart and when the websocket was reconnected, leading to some stalled initial message replies. We delete the logic that sends pending messages on shutdown_reply, since those pending messages will be more correctly sent when the websocket reconnects anyway. We also don’t worry about setting the kernel session there since the calling function does that logic.
|
@davidbrochart - can you check the binder link generated at #9571 (or test that PR directly) to see if it solves the problem you saw here? Testing locally, that fix seemed to solve the issues I was seeing. |
|
I tried the generated binder, and I could see once that all the cells were marked with |
Let's take the review conversation over to that PR. I'll copy your comment over there. |
|
Hi, I have just read the various issues around this and wonder if the following can help
|
|
Thanks for looking at this @echarles. Can we distinguish between what might be a good idea in the future, and what is needed now to fix this rather serious bug without making things worse? To me, it sounds like your (1) item may be something to address in the future, and (2) may be a suggestion for a change on this PR? Am I reading things how you intended? That does seem elegant to always have a queue, and always work through that queue. Except, of course, for our queue-jumping kernel_info messages that we need right now to clear the restarting status sentinel. But I think the logic would look very similar to how it does now? Right now, if we send a message, basically the _sendMessage function deals with whether to put it on a queue, or to send immediately - that is transparent to the user. Even if we enqueued all messages (except those special kernel_info_request ones), somewhere we would need logic to flush the queue. I'd hate to put that on to the user to do as an extra step, so we'd have something somewhere keeping track of the queue state, and when a new message is added, determining whether we should flush the queue or not - which is essentially what _sendMessage is doing right now anyway. In other words, we'll need the same logic as we have now, and the api would also essentially look the same, especially to the user. Am I missing something? |
Yes, (1) is for longer term, (2) can be discussed in the scope of the needed hot fix.
I don' know but I think I do miss something as I have not spent enough time to look at all the conditions and the conversation history. I just had a feeling that a managed queue could help solve the issues. When I look at jupyterlab/packages/services/src/terminal/default.ts Lines 102 to 114 in 9ddc785 _sendPending before sending the message.
|
Well, you have to be a bit more careful - _sendPending calls this function as the lowest-level sending function IIRC, so you have to guard against the queue flush sends getting cycled back onto the queue itself. The other thing is that it wasn't clear this was a problem in kernels before the restart status logic was introduced. The connection status change triggered a queue flush, which IIRC was synchronous, which wouldn't let other messages jump the queue. If that is the case, it may not be a problem here either. |
|
(Let's please take discussion of the specific PR over to the PR, and leave more general conversation here, so conversation about that specific fix isn't scattered on other issues) |
github-actions
bot
added
the
status:resolved-locked
Description
Manually executing cells while the kernel is restarting leads to out-of-order cell execution:
Reproduce
*, and/or some cells executed out of order, potentially leading to execution errors.Expected behavior
Cells should be executed in order.
Context
Troubleshoot Output
jovyan@jupyter-jupyter-2dwidgets-2dipyleaflet-2duyxyynfv:~$ jupyter troubleshoot $PATH: /srv/conda/envs/notebook/bin /srv/conda/condabin /home/jovyan/.local/bin /home/jovyan/.local/bin /srv/conda/envs/notebook/bin /srv/conda/bin /srv/npm/bin /usr/local/sbin /usr/local/bin /usr/sbin /usr/bin /sbin /binsys.path:
/srv/conda/envs/notebook/bin
/srv/conda/envs/notebook/lib/python37.zip
/srv/conda/envs/notebook/lib/python3.7
/srv/conda/envs/notebook/lib/python3.7/lib-dynload
/srv/conda/envs/notebook/lib/python3.7/site-packages
sys.executable:
/srv/conda/envs/notebook/bin/python
sys.version:
3.7.8 | packaged by conda-forge | (default, Jul 23 2020, 03:54:19)
[GCC 7.5.0]
platform.platform():
Linux-4.19.112+-x86_64-with-debian-buster-sid
which -a jupyter:
/srv/conda/envs/notebook/bin/jupyter
/srv/conda/envs/notebook/bin/jupyter
pip list:
Package Version
----------------------- -------------------
affine 2.3.0
alembic 1.4.2
anyio 2.0.2
async-generator 1.10
attrs 19.3.0
Babel 2.9.0
backcall 0.2.0
bleach 3.1.5
blinker 1.4
bqplot 0.12.18
branca 0.4.2
brotlipy 0.7.0
certifi 2020.6.20
certipy 0.1.3
cffi 1.14.0
cftime 1.3.0
chardet 3.0.4
click 7.1.2
click-plugins 1.1.1
cligj 0.7.1
cryptography 3.0
cycler 0.10.0
decorator 4.4.2
defusedxml 0.6.0
entrypoints 0.3
Fiona 1.8.18
Flask 1.1.2
GDAL 3.1.4
geopandas 0.8.1
idna 2.10
importlib-metadata 1.7.0
ipykernel 5.3.4
ipyleaflet 0.13.5
ipython 7.16.1
ipython-genutils 0.2.0
ipywidgets 7.6.2
itsdangerous 1.1.0
jedi 0.17.2
Jinja2 2.11.2
json5 0.9.4
jsonschema 3.2.0
jupyter-client 6.1.6
jupyter-core 4.6.3
jupyter-offlinenotebook 0.1.0
jupyter-server 1.1.4
jupyter-telemetry 0.0.5
jupyterhub 1.1.0
jupyterlab 3.0.1
jupyterlab-server 2.0.0
jupyterlab-widgets 1.0.0
kiwisolver 1.3.1
Mako 1.1.0
MarkupSafe 1.1.1
matplotlib 3.3.3
mistune 0.8.4
munch 2.5.0
nbclassic 0.2.5
nbconvert 5.6.1
nbformat 5.0.7
nbresuse 0.3.3
netCDF4 1.5.5.1
notebook 6.0.3
nteract-on-jupyter 2.1.3
numpy 1.19.4
oauthlib 3.0.1
olefile 0.46
packaging 20.4
pamela 1.0.0
pandas 1.2.0
pandocfilters 1.4.2
parso 0.7.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.1.0
pip 20.1.1
prometheus-client 0.8.0
prompt-toolkit 3.0.5
psutil 5.7.2
ptyprocess 0.6.0
pycparser 2.20
pycurl 7.43.0.5
Pygments 2.6.1
PyJWT 1.7.1
pyOpenSSL 19.1.0
pyparsing 2.4.7
pyproj 2.6.1.post1
PyQt5 5.12.3
PyQt5-sip 4.19.18
PyQtChart 5.12
PyQtWebEngine 5.12.1
pyrsistent 0.16.0
PySocks 1.7.1
python-dateutil 2.8.1
python-editor 1.0.4
python-json-logger 0.1.11
pytz 2020.5
pyzmq 19.0.1
rasterio 1.1.8
requests 2.24.0
Rtree 0.9.7
ruamel.yaml 0.16.6
ruamel.yaml.clib 0.2.0
scipy 1.6.0
Send2Trash 1.5.0
setuptools 49.2.0.post20200712
Shapely 1.7.1
six 1.15.0
sniffio 1.2.0
snuggs 1.4.7
SQLAlchemy 1.3.18
terminado 0.8.3
testpath 0.4.4
tornado 6.1
tqdm 4.55.1
traitlets 4.3.3
traittypes 0.2.1
typing-extensions 3.7.4.3
urllib3 1.25.10
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 1.0.1
wheel 0.34.2
widgetsnbextension 3.5.1
xarray 0.16.2
zipp 3.1.0
conda list:
# packages in environment at /srv/conda/envs/notebook:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 0_gnu conda-forge
affine 2.3.0 py_0 conda-forge
alembic 1.4.2 pyh9f0ad1d_0 conda-forge
anyio 2.0.2 py37h89c1867_3 conda-forge
async_generator 1.10 py_0 conda-forge
attrs 19.3.0 py_0 conda-forge
babel 2.9.0 pyhd3deb0d_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
bleach 3.1.5 pyh9f0ad1d_0 conda-forge
blinker 1.4 py_1 conda-forge
boost-cpp 1.72.0 h9359b55_3 conda-forge
bqplot 0.12.18 pyh9f0ad1d_0 conda-forge
branca 0.4.2 pyhd8ed1ab_0 conda-forge
brotlipy 0.7.0 py37h8f50634_1000 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2020.6.20 hecda079_0 conda-forge
cairo 1.16.0 h488836b_1006 conda-forge
certifi 2020.6.20 py37hc8dfbb8_0 conda-forge
certipy 0.1.3 py_0 conda-forge
cffi 1.14.0 py37hd463f26_0 conda-forge
cfitsio 3.470 hb418390_7 conda-forge
cftime 1.3.0 py37ha21ca33_0 conda-forge
chardet 3.0.4 py37hc8dfbb8_1006 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.1 pyhd8ed1ab_0 conda-forge
cryptography 3.0 py37hb09aad4_0 conda-forge
curl 7.71.1 he644dc0_3 conda-forge
cycler 0.10.0 py_2 conda-forge
dbus 1.13.18 hb2f20db_0 defaults
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
entrypoints 0.3 py37hc8dfbb8_1001 conda-forge
expat 2.2.10 he6710b0_2 defaults
fiona 1.8.18 py37h527b4ca_0 conda-forge
flask 1.1.2 pyh9f0ad1d_0 conda-forge
fontconfig 2.13.1 h1056068_1002 conda-forge
freetype 2.10.4 he06d7ca_0 conda-forge
freexl 1.0.6 h27cfd23_0 defaults
gdal 3.1.4 py37h2b22b9e_0 conda-forge
geopandas 0.8.1 py_0 conda-forge
geos 3.8.1 he1b5a44_0 conda-forge
geotiff 1.6.0 h5d11630_3 conda-forge
gettext 0.19.8.1 hf34092f_1004 conda-forge
giflib 5.2.1 h516909a_2 conda-forge
glib 2.66.3 h58526e2_0 conda-forge
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
hdf4 4.2.13 h10796ff_1004 conda-forge
hdf5 1.10.6 nompi_h6a2412b_1113 conda-forge
icu 67.1 he1b5a44_0 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
importlib-metadata 1.7.0 py37hc8dfbb8_0 conda-forge
importlib_metadata 1.7.0 0 conda-forge
ipykernel 5.3.4 py37h43977f1_0 conda-forge
ipyleaflet 0.13.5 pyhd8ed1ab_0 conda-forge
ipython 7.16.1 py37h43977f1_0 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.2 pyhd3deb0d_1 conda-forge
itsdangerous 1.1.0 py_0 conda-forge
jedi 0.17.2 py37hc8dfbb8_0 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
jpeg 9d h516909a_0 conda-forge
json-c 0.13.1 hbfbb72e_1002 conda-forge
json5 0.9.4 pyh9f0ad1d_0 conda-forge
jsonschema 3.2.0 py37hc8dfbb8_1 conda-forge
jupyter-offlinenotebook 0.1.0 pypi_0 pypi
jupyter_client 6.1.6 py_0 conda-forge
jupyter_core 4.6.3 py37hc8dfbb8_1 conda-forge
jupyter_server 1.1.4 py37h89c1867_0 conda-forge
jupyter_telemetry 0.0.5 py_0 conda-forge
jupyterhub-base 1.1.0 py37_2 conda-forge
jupyterhub-singleuser 1.1.0 py37_2 conda-forge
jupyterlab 3.0.1 pyhd8ed1ab_0 conda-forge
jupyterlab_server 2.0.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
kealib 1.4.14 h0042707_0 conda-forge
kiwisolver 1.3.1 py37hc928c03_0 conda-forge
krb5 1.17.1 hfafb76e_1 conda-forge
lcms2 2.11 hcbb858e_1 conda-forge
ld_impl_linux-64 2.34 h53a641e_7 conda-forge
libblas 3.9.0 6_openblas conda-forge
libcblas 3.9.0 6_openblas conda-forge
libclang 10.0.1 default_hde54327_1 conda-forge
libcurl 7.71.1 hcdd3856_3 conda-forge
libdap4 3.20.6 h1d1bd15_1 conda-forge
libedit 3.1.20191231 h46ee950_1 conda-forge
libevent 2.1.10 hcdb4288_3 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h5dbcf3e_17 conda-forge
libgdal 3.1.4 h670eac6_0 conda-forge
libgfortran-ng 9.3.0 he4bcb1c_17 conda-forge
libgfortran5 9.3.0 he4bcb1c_17 conda-forge
libglib 2.66.3 hbe7bbb4_0 conda-forge
libgomp 9.3.0 h5dbcf3e_17 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 hd79254b_1012 conda-forge
liblapack 3.9.0 6_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libnetcdf 4.7.4 nompi_h56d31a8_107 conda-forge
libopenblas 0.3.12 pthreads_h4812303_1 conda-forge
libpng 1.6.37 hed695b0_2 conda-forge
libpq 12.3 hfd2b0eb_3 conda-forge
libsodium 1.0.17 h516909a_0 conda-forge
libspatialindex 1.9.3 he1b5a44_3 conda-forge
libspatialite 5.0.0 h4dde289_0 conda-forge
libssh2 1.9.0 hab1572f_4 conda-forge
libstdcxx-ng 9.3.0 h2ae2ef3_17 conda-forge
libtiff 4.2.0 hdc55705_0 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libuv 1.39.0 h516909a_0 conda-forge
libwebp-base 1.1.0 h516909a_3 conda-forge
libxcb 1.14 h7b6447c_0 defaults
libxkbcommon 0.10.0 he1b5a44_0 conda-forge
libxml2 2.9.10 h68273f3_2 conda-forge
lz4-c 1.9.2 he1b5a44_3 conda-forge
mako 1.1.0 py_0 conda-forge
markupsafe 1.1.1 py37h8f50634_1 conda-forge
matplotlib 3.3.3 py37h89c1867_0 conda-forge
matplotlib-base 3.3.3 py37h4f6019d_0 conda-forge
mistune 0.8.4 py37h8f50634_1001 conda-forge
munch 2.5.0 py_0 conda-forge
mysql-common 8.0.21 2 conda-forge
mysql-libs 8.0.21 hf3661c5_2 conda-forge
nbclassic 0.2.5 pyhd8ed1ab_0 conda-forge
nbconvert 5.6.1 py37hc8dfbb8_1 conda-forge
nbformat 5.0.7 py_0 conda-forge
nbresuse 0.3.3 py_0 conda-forge
ncurses 6.2 he1b5a44_1 conda-forge
netcdf4 1.5.5.1 nompi_py37hdbee05a_100 conda-forge
nodejs 14.11.0 h568c755_0 conda-forge
notebook 6.0.3 py37hc8dfbb8_1 conda-forge
nspr 4.29 he1b5a44_1 conda-forge
nss 3.60 hb5efdd6_0 conda-forge
nteract_on_jupyter 2.1.3 py_0 conda-forge
numpy 1.19.4 py37haa41c4c_2 conda-forge
oauthlib 3.0.1 py_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.3.1 hf7af979_3 conda-forge
openssl 1.1.1i h7f98852_0 conda-forge
packaging 20.4 pyh9f0ad1d_0 conda-forge
pamela 1.0.0 py_0 conda-forge
pandas 1.2.0 py37hdc94413_0 conda-forge
pandoc 2.10.1 h516909a_0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parso 0.7.1 pyh9f0ad1d_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pexpect 4.8.0 py37hc8dfbb8_1 conda-forge
pickleshare 0.7.5 py37hc8dfbb8_1001 conda-forge
pillow 8.1.0 py37he6b4880_0 conda-forge
pip 20.1.1 py_1 conda-forge
pixman 0.38.0 h516909a_1003 conda-forge
poppler 0.89.0 h4190859_1 conda-forge
poppler-data 0.4.10 0 conda-forge
postgresql 12.3 h6303168_3 conda-forge
proj 7.1.1 h966b41f_3 conda-forge
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge
prompt-toolkit 3.0.5 py_1 conda-forge
psutil 5.7.2 py37h8f50634_0 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pycurl 7.43.0.5 py37hce7685b_2 conda-forge
pygments 2.6.1 py_0 conda-forge
pyjwt 1.7.1 py_0 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyproj 2.6.1.post1 py37h6415a23_3 conda-forge
pyqt 5.12.3 py37h89c1867_6 conda-forge
pyqt-impl 5.12.3 py37he336c9b_6 conda-forge
pyqt5-sip 4.19.18 py37hcd2ae1e_6 conda-forge
pyqtchart 5.12 py37he336c9b_6 conda-forge
pyqtwebengine 5.12.1 py37he336c9b_6 conda-forge
pyrsistent 0.16.0 py37h8f50634_0 conda-forge
pysocks 1.7.1 py37hc8dfbb8_1 conda-forge
python 3.7.8 h6f2ec95_0_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python-editor 1.0.4 py_0 conda-forge
python-json-logger 0.1.11 py_0 conda-forge
python_abi 3.7 1_cp37m conda-forge
pytz 2020.5 pyhd8ed1ab_0 conda-forge
pyzmq 19.0.1 py37hac76be4_0 conda-forge
qt 5.12.9 h1f2b2cb_0 conda-forge
rasterio 1.1.8 py37h527b4ca_0 conda-forge
readline 8.0 he28a2e2_2 conda-forge
requests 2.24.0 pyh9f0ad1d_0 conda-forge
rtree 0.9.7 py37h0b55af0_0 conda-forge
ruamel.yaml 0.16.6 py37h8f50634_1 conda-forge
ruamel.yaml.clib 0.2.0 py37h8f50634_1 conda-forge
scipy 1.6.0 py37h14a347d_0 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 49.2.0 py37hc8dfbb8_0 conda-forge
shapely 1.7.1 py37hedb1597_1 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
sniffio 1.2.0 py37hc8dfbb8_0 conda-forge
snuggs 1.4.7 py_0 conda-forge
sqlalchemy 1.3.18 py37h8f50634_0 conda-forge
sqlite 3.34.0 h74cdb3f_0 conda-forge
terminado 0.8.3 py37hc8dfbb8_1 conda-forge
testpath 0.4.4 py_0 conda-forge
tiledb 2.1.5 h17508cd_0 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
tornado 6.1 py37h4abf009_0 conda-forge
tqdm 4.55.1 pyhd8ed1ab_0 conda-forge
traitlets 4.3.3 py37hc8dfbb8_1 conda-forge
traittypes 0.2.1 pyh9f0ad1d_2 conda-forge
typing_extensions 3.7.4.3 py_0 conda-forge
tzcode 2020f h7f98852_0 conda-forge
urllib3 1.25.10 py_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
werkzeug 1.0.1 pyh9f0ad1d_0 conda-forge
wheel 0.34.2 py_1 conda-forge
widgetsnbextension 3.5.1 py37hc8dfbb8_1 conda-forge
xarray 0.16.2 pyhd8ed1ab_0 conda-forge
xerces-c 3.2.3 hfe33f54_1 conda-forge
xorg-kbproto 1.0.7 h14c3975_1002 conda-forge
xorg-libice 1.0.10 h516909a_0 conda-forge
xorg-libsm 1.2.3 h84519dc_1000 conda-forge
xorg-libx11 1.6.12 h516909a_0 conda-forge
xorg-libxext 1.3.4 h516909a_0 conda-forge
xorg-libxrender 0.9.10 h516909a_1002 conda-forge
xorg-renderproto 0.11.1 h14c3975_1002 conda-forge
xorg-xextproto 7.3.0 h14c3975_1002 conda-forge
xorg-xproto 7.0.31 h14c3975_1007 conda-forge
xz 5.2.5 h516909a_1 conda-forge
zeromq 4.3.2 he1b5a44_2 conda-forge
zipp 3.1.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.4.8 hdf46e1d_0 conda-forge
Browser Output
243.d9d134ad2ffdfb9205b3.js:2 Deprecated include of L.Mixin.Events: this property will be removed in future releases, please inherit from L.Evented instead. Error at https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/243.d9d134ad2ffdfb9205b3.js:2:3143 at Function.C.extend (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/243.d9d134ad2ffdfb9205b3.js:2:3162) at Object.665 (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/665.3328989849f89dbc115a.js:1:4763) at j (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/remoteEntry.d7ecc23c8a9ea32680f5.js:1:795) at https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/remoteEntry.d7ecc23c8a9ea32680f5.js:1:3893 at Object.w. (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/remoteEntry.d7ecc23c8a9ea32680f5.js:1:9876) at j (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/remoteEntry.d7ecc23c8a9ea32680f5.js:1:795) at Object.8197 (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/138.cfc773cf0b77045b9fbb.js:1:59505) at j (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/remoteEntry.d7ecc23c8a9ea32680f5.js:1:795) at Module.4795 (https://hub.gke2.mybinder.org/user/jupyter-widgets-ipyleaflet-uyxyynfv/lab/extensions/jupyter-leaflet/static/138.cfc773cf0b77045b9fbb.js:1:43381) (anonymous) @ 243.d9d134ad2ffdfb9205b3.js:2 jlab_core.3099b1be2bd042406f2c.js:2 Ignored setting registry preload errors. Array(1) (anonymous) @ jlab_core.3099b1be2bd042406f2c.js:2 jlab_core.3099b1be2bd042406f2c.js:2 Ignored setting registry preload errors. Array(1) (anonymous) @ jlab_core.3099b1be2bd042406f2c.js:2 jlab_core.3099b1be2bd042406f2c.js:2 Connection lost, reconnecting in 0 seconds. _reconnect @ jlab_core.3099b1be2bd042406f2c.js:2 static/components/MathJax/MathJax.js?config=TeX-AMS-MML_HTMLorMML-full,Safe&delayStartupUntil=configured:1 Uncaught SyntaxError: Unexpected token '<' jlab_core.3099b1be2bd042406f2c.js:2 Uncaught ReferenceError: MathJax is not defined at i._onLoad (jlab_core.3099b1be2bd042406f2c.js:2) at HTMLScriptElement. (jlab_core.3099b1be2bd042406f2c.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Connection lost, reconnecting in 0 seconds. _reconnect @ jlab_core.3099b1be2bd042406f2c.js:2 jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n.o (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._propagateEvent (243.d9d134ad2ffdfb9205b3.js:2) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n.o (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._propagateEvent (243.d9d134ad2ffdfb9205b3.js:2) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n.o (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Connection lost, reconnecting in 0 seconds. _reconnect @ jlab_core.3099b1be2bd042406f2c.js:2 jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n.o (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Ee.t.send (272.2a8425db7209008188fc.js:1) at Ie.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._propagateEvent (243.d9d134ad2ffdfb9205b3.js:2) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Uncaught Error: Cannot send at l.send (jlab_core.3099b1be2bd042406f2c.js:2) at e.send (272.2a8425db7209008188fc.js:1) at Yt.t.send (272.2a8425db7209008188fc.js:1) at Xt.t.send (272.2a8425db7209008188fc.js:1) at n. (138.cfc773cf0b77045b9fbb.js:1) at n.fire (243.d9d134ad2ffdfb9205b3.js:2) at n._fireDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at n._handleDOMEvent (243.d9d134ad2ffdfb9205b3.js:2) at HTMLDivElement.r (243.d9d134ad2ffdfb9205b3.js:2) jlab_core.3099b1be2bd042406f2c.js:2 Connection lost, reconnecting in 0 seconds. _reconnect @ jlab_core.3099b1be2bd042406f2c.js:2 jlab_core.3099b1be2bd042406f2c.js:2 Connection lost, reconnecting in 0 seconds. _reconnect @ jlab_core.3099b1be2bd042406f2c.js:2The text was updated successfully, but these errors were encountered: