Memory leak in tqdm_notebook

[rfriel]

In some cases, using the tqdm_notebook class to iterate through an object multiple times will accumulate copies of the object in memory.

This causes the user to run out of memory when, for example, using tqdm_notebook to iterate through a large dataset once per epoch in neural net training.

The root cause is the reference cycle created on this line.

I have written a full walkthrough in this Colab notebook. I recommend reading the notebook, as it shows the memory issue directly using tracemalloc, but I'll reproduce the examples here.

Versions:

• the problem exists in 4.59.0 (my local version) and 4.62.0 (latest)
• the problem does not exist in 4.41.1 (default version on Colab)

Related issues:

The only issue I could find with a memory leak was #746 with pytorch's DataLoader. However, that issue involved non-notebook tqdm.

(It's worth nothing, though, that I first experienced the issue when using DataLoader.)

Minimal example

(uses numpy to create a large array quickly)

In the loop below, one copy of y's memory cost is added per loop iteration. For example, the loop will crash a standard Colab runtime (~12GB memory).

from copy import deepcopy
import numpy as np
from tqdm.notebook import tqdm

y = np.ones((80000, 3000))

def loop_fn(y_):
    # deepcopy ensures we allocate new memory in each function call
    # (which should be released when the function returns)
    y2 = deepcopy(y_)
    for _ in tqdm(y2):
        pass

for i in range(10):
    loop_fn(y)

Ruling out causes besides tqdm_notebook

The issue does not occur if the import above is replaced with from tqdm import tqdm.

The issue does not occur if we replace loop_fn with

def loop_fn(y_):
    y2 = deepcopy(y_)

so merely copying the array (without tqdm) is fine.

Root cause in tqdm_notebook

The root cause is the line

self.container.pbar = self

which creates a reference cycle between self and self.container.

Thus, the garbage collector cannot collect a tqdm_notebook instance via reference counting. So, for example, the instance does not get collected immediately upon leaving a function like loop_fn above.

This prevents garbage collection of its iterable attribute (set to y2 above).

To see that this is the problem, observe that this does not leak memory:

def loop_fn_fixed(y_):
    y2 = deepcopy(y_)
    iterable = tqdm_notebook(y2)
    for __ in iterable:
        pass
    del iterable.container.pbar  # fix!

for i in range(10):
    loop_fn_fixed(y)

while an identical loop_fn_fixed without the last line does loop memory.

---

• I have marked all applicable categories:
  • exception-raising bug
  • visual output bug
• I have visited the source website, and in particular
  read the known issues
• I have searched through the issue tracker for duplicates
• I have mentioned version numbers, operating system and
  environment, where applicable:

  import tqdm, sys
  print(tqdm.__version__, sys.version, sys.platform)

[casperdcl]

Can you try with devel (pip install "git+https://github.com/tqdm/tqdm@devel#egg=tqdm")?

[rfriel]

Can you try with devel (pip install "git+https://github.com/tqdm/tqdm@devel#egg=tqdm")?

I tried it and it no longer leaks. Thanks!