Keras progress bars cause jupyterlab to freeze firefox and chrome

[ogrisel]

I noticed the builtin keras progress bar often causes the jupyterlab UI to freeze after a while (both under firefox and chrome, under Linux at least). I do not observe this issue under the traditional jupyter notebook app.

Here is a minimal reproduction script that does not depend on Keras. I am not sure Keras is implemented that way but might script seem to be able able to reproduce the same behavior as Keras is causing.

import time


def make_progress(n_steps=int(1e4), line_length=70, delay=0,
                  flush_every=1e-2, prompt=""):
    last_flush = time.time()
    for i in range(n_steps):
        progressed = ((i + 1) * line_length) // n_steps
        prefix = '#' * progressed
        suffix = '-' * (line_length - progressed)
        now = time.time()
        flush = (now - last_flush) > flush_every
        if flush:
            last_flush = now
        print(f'\r{prompt}{i + 1:05d} [{prefix}{suffix}]', end="", flush=flush)
        if delay:
            time.sleep(delay)

            
for i in range(30):
    make_progress(prompt=f"{i + 1:02d} ")
    print()

Steps to reproduce:

• copy the above snippet in a jupyterlab cell in a fresh new Python 3.6 notebook
• execute the cell

Observed:

• the progress bars display one after the other;
• the first progress bars are very smooth;
• after 5 or 10 bars, progression is much less smooth;
• after 20 or so firefox window is completely frozen: it's clicking on the interrupt button takes a while to be taken into account.

Expected:

• smooth progress bars as observed when running the same notebook in the old jupyter notebook instead of jupyterlab.

I suspect that the \r not handled in the same way as in jupyter notebook and some datastructure in the browser is growing beyond acceptable.

Note that this bug is not the same as #1104 that was already fixed in previous versions of jupyter lab.

This issue was observed using jupyter lab 0.31.10. I have just upgraded to 0.31.12 and it seems to behave a bit better: I now get the following warning after a while:

IOPub data rate exceeded.
The notebook server will temporarily stop sending output
to the client in order to avoid crashing it.
To change this limit, set the config variable
`--NotebookApp.iopub_data_rate_limit`.

Current values:
NotebookApp.iopub_data_rate_limit=1000000.0 (bytes/sec)
NotebookApp.rate_limit_window=3.0 (secs)

but firefox still gets much slower than under the original jupyter notebook app after a while.

[ogrisel]

Actually, I also observed the IOPub data rate exceeded. warning on jupyter notebook.

[ogrisel]

Maybe it's not that a big issue anymore in the recent versions. There might still be opportunities for further optimization of transfer of text streams with frequent calls to flush that also have many \r occurrences. Maybe using a server-side buffer in case of too frequent calls to flush for instance.

[jasongrout]

Maybe it's not that a big issue anymore in the recent versions.

Thanks. So are you saying JupyterLab behaves like the classic notebook right now?

The server-side buffer idea would be an issue for the ipython kernel or the jupyter notebook server.

[ogrisel]

Yes I believe so but both seem to slow down when running the above snippet.

[sccolbert]

but firefox still gets much slower than under the original jupyter notebook app after a while.

This is not too surprising. The updates are likely causing a layout reflow of the notebook, which we know is slow on FF and will be fixed for 1.0

[jasongrout]

For what it's worth, I tried the example above in current jlab master and didn't see any slowdown across the 30 progress bars. FF 59 on macOS 10.13.3.

[jasongrout]

cf also #1639 for firefox rendering/layout slowdown issues.

[MrMathias]

I am also experiencing freezes using chrome/windows. Does this have to do with the update rate or the absolute number of updates of the bar? - I have also experienced that the bar itself just stops progressing, might be related.

[uzi0espil]

I am facing same issue.. Running a keras script on Chrome browser v. 67.0.3396.99, After some epochs, the browser hangs and asks for killing the page. I tried also with incognite and still same problem

Jupyterlab version: 0.32.1
OS: Ubuntu 16.04

[wernight]

Could have a bit less output, but not nothing like verbose=0. This may "solve" it for most cases.

[sebastianffx]

Same issue here. Ubuntu 16.04, Chromium 70.0.3, Jupyterlab 0.35.4, Keras 2.2, python 3.6
My workaround so far is just to check that training starts OK for some epochs and then stop it, download the notebook as a python script, and execute it directly on the terminal.

[atamborrino]

Same issue here. I can not train a Keras model for more than 5 epochs... IMHO this is a blocking issue for using JupyterLab with Keras.

[ghostandthemachine]

I've had this same problem. My workaround was to set verbose=0 in the fit call then also add a LambdaCallback which updates an IPythonWidget IntProgress bar so I get the same info. I think this is actually a nicer solution then the long print statements. I pass the display elements to my callback creation fn:

def _progress_cb(progress, html):
    
    def on_epoch_end(batch, logs):
        progress.value += 1
        html.value = f"<div><div>Epoch: {batch}</div><div>{logs}</div></div>"
    
    return callbacks.LambdaCallback(
        on_epoch_end=on_epoch_end
    )

Then use it with something like:

html = widgets.HTML(value="")
progress = widgets.IntProgress(min=0, max=num_total_epochs, value=0)

display(progress)
display(html)

model.fit(x=x, 
          y=y,
          verbose=0,
          callbacks=[_progress_cb(progress, html)])

You obviously loose history in your cell output div but I find TensorBoard more useful for that anyway.

[jasongrout]

I've had this same problem. My workaround was to set verbose=0 in the fit call then also add a LambdaCallback which updates an IPythonWidget IntProgress bar so I get the same info. I think this is actually a nicer solution then the long print statements.

Nice! Thanks for sharing!

[dbuades]

Same issue here, though it works perfectly with the classical jupyter notebook.

[divamgupta]

Hi, any updates regarding this issue?
The initial versions of jupyterlab did not have this issue

[jasongrout]

Is this still an issue with the 1.0a3 released last week? There has been some recent work on handling \r, which is what these progress bars are using, IIRC.

[jasongrout]

Yes, this is still an issue in 1.0a3. The code in the OP works great for me in classic notebook, ff 66, macOS. Stops dead in the first progress bar in JupyterLab 1.0a3, ff 66 or chrome 74.

I'm going to tag it for 1.0 and try to spend at least some time looking at it, to see if there is an obvious optimization we're missing somewhere.

[jasongrout]

@ian-r-rose has some thoughts on this, like that the classic notebook does some update throttling.

[ian-r-rose]

Specifically here

[blink1073]

Thanks to @ian-r-rose for the fix! I just released the fix in 0.35.6. I verified by running @ogrisel's original example.