Memory leak when opening images

[aIligat0r]

Hello!
I have not found a solution to this problem. If there was a decision, then I apologize in advance.

When opening images, the amount of memory is constantly increasing. The library somehow saves the open images. I'm opening an image in context.

Simulated leak:

import os
import tracemalloc

import psutil
from PIL import Image


def main():
    with open("/tmp/github.png", "rb") as img_file:
        with Image.open(fp=img_file) as img:
            pass


if __name__ == '__main__':
    print("PIL version:", PIL.__version__)
    tracemalloc.start()
    pid = os.getpid()
    process = psutil.Process(pid)
    for i in range(100):
        main()
        snapshot = tracemalloc.take_snapshot()
        stats = snapshot.statistics("lineno")
        for s in stats[:10]:
            print(s.size, s.traceback)
        memory_info = process.memory_info()
        print(f'RAM usage: {memory_info.rss / 1000 / 1000} MB')
        print(f"Replay: {i+1}\n" + "*" * 70)
    tracemalloc.stop()

Out:

PIL version: 10.3.0

882738 <frozen importlib._bootstrap_external>:729
34816 /usr/lib/python3.11/tracemalloc.py:67
30464 /usr/lib/python3.11/tracemalloc.py:505
26400 /usr/lib/python3.11/tracemalloc.py:558
26160 /usr/lib/python3.11/tracemalloc.py:498
7727 <frozen importlib._bootstrap>:241
5286 <frozen importlib._bootstrap_external>:128
4788 /usr/lib/python3.11/enum.py:532
4408 /usr/lib/python3.11/tracemalloc.py:534
4228 /home/user/venv/lib/python3.11/site-packages/PIL/PngImagePlugin.py:700
RAM usage: 21.614592000000002 MB
Replay: 4
==================================================================
882738 <frozen importlib._bootstrap_external>:729
34816 /usr/lib/python3.11/tracemalloc.py:67
30296 /usr/lib/python3.11/tracemalloc.py:505
26016 /usr/lib/python3.11/tracemalloc.py:498
25536 /usr/lib/python3.11/tracemalloc.py:193
7727 <frozen importlib._bootstrap>:241
5286 <frozen importlib._bootstrap_external>:128
4788 /usr/lib/python3.11/enum.py:532
4392 /usr/lib/python3.11/tracemalloc.py:534
4228 /home/user/venv/lib/python3.11/site-packages/PIL/PngImagePlugin.py:700
RAM usage: 21.614592000000002 MB
Replay: 5
==================================================================
882738 <frozen importlib._bootstrap_external>:729
34816 /usr/lib/python3.11/tracemalloc.py:67
30352 /usr/lib/python3.11/tracemalloc.py:505
26016 /usr/lib/python3.11/tracemalloc.py:498
25536 /usr/lib/python3.11/tracemalloc.py:193
7727 <frozen importlib._bootstrap>:241
5286 <frozen importlib._bootstrap_external>:128
4788 /usr/lib/python3.11/enum.py:532
4392 /usr/lib/python3.11/tracemalloc.py:534
4228 /home/user/venv/lib/python3.11/site-packages/PIL/PngImagePlugin.py:700
RAM usage: 21.745664 MB
Replay: 6
==================================================================

...


880946 <frozen importlib._bootstrap_external>:729
34880 /usr/lib/python3.11/tracemalloc.py:67
30464 /usr/lib/python3.11/tracemalloc.py:505
26112 /usr/lib/python3.11/tracemalloc.py:498
15812 /home/user/venv/lib/python3.11/site-packages/PIL/PngImagePlugin.py:192
13248 /usr/lib/python3.11/tracemalloc.py:193
9344 /usr/lib/python3.11/tracemalloc.py:558
7727 <frozen importlib._bootstrap>:241
5286 <frozen importlib._bootstrap_external>:128
4788 /usr/lib/python3.11/enum.py:532
RAM usage: 21.876736 MB
Replay: 98
==================================================================
880946 <frozen importlib._bootstrap_external>:729
34880 /usr/lib/python3.11/tracemalloc.py:67
30408 /usr/lib/python3.11/tracemalloc.py:505
26064 /usr/lib/python3.11/tracemalloc.py:498
19488 /usr/lib/python3.11/tracemalloc.py:193
15871 /home/user/venv/lib/python3.11/site-packages/PIL/PngImagePlugin.py:192
7727 <frozen importlib._bootstrap>:241
5286 <frozen importlib._bootstrap_external>:128
4788 /usr/lib/python3.11/enum.py:532
4408 /usr/lib/python3.11/tracemalloc.py:534
RAM usage: 21.876736 MB
Replay: 99
==================================================================
880946 <frozen importlib._bootstrap_external>:729
34880 /usr/lib/python3.11/tracemalloc.py:67
30464 /usr/lib/python3.11/tracemalloc.py:505
26112 /usr/lib/python3.11/tracemalloc.py:498
15930 /home/user/venv/lib/python3.11/site-packages/PIL/PngImagePlugin.py:192
12512 /usr/lib/python3.11/tracemalloc.py:558
10032 /usr/lib/python3.11/tracemalloc.py:193
7727 <frozen importlib._bootstrap>:241
5286 <frozen importlib._bootstrap_external>:128
4788 /usr/lib/python3.11/enum.py:532
RAM usage: 21.876736 MB
Replay: 100
==================================================================

It looks like a memory leak in the plugin PngImagePlugin.py

If there is a solution how to get around this problem in tasks where you have to open a lot of images?

I have previously encountered an error in Webp formats. The leak was in the WebPImagePlugin plugin. I solved this by changing the value of the variable HAVE_WEBPANIM (PIL._webp.HAVE_WEBPANIM) to False. But now I'm facing a problem in PngImagePlugin

[Yay295]

It looks like it thinks the errors might be coming from lines 192 and 700?

Pillow/src/PIL/PngImagePlugin.py

Lines 188 to 192 in 955c5da

	def call(self, cid, pos, length):
	"""Call the appropriate chunk handler"""
	logger.debug("STREAM %r %s %s", cid, pos, length)
	return getattr(self, "chunk_" + cid.decode("ascii"))(pos, length)

Pillow/src/PIL/PngImagePlugin.py

Lines 700 to 702 in 955c5da

	class PngImageFile(ImageFile.ImageFile):
	format = "PNG"
	format_description = "Portable network graphics"

[radarhere]

If I increase the number of loops to 1000, I find that the memory drops down again at a certain point. Since it is not continuously increasing, I don't think it is a leak.

You might like to read #7935, in particular #7935 (comment)

Pillow's memory allocator doesn't necessarily release the memory in the pool back as soon as an image is destroyed, as it uses that memory pool for future allocations. See Storage.c (https://github.com/python-pillow/Pillow/blob/main/src/libImaging/Storage.c#L310) for the implementation.

[Yay295]

If I increase the number of loops to 1000, I find that the memory drops down again at a certain point.

It would probably be good to add gc.collect() (import gc) to the end of each loop. It might just not be running that often.