Massive memory leak since 8.5.5 / leaking the entire stack frame and all local variables everywhere

[luk2302]

In #187 a change was implemented that included the current stack frame in an internal cache, therefore causing all local variables of the entire call stack to never be freed. Demo:

from dataclasses import dataclass
import marshmallow_dataclass
from marshmallow.schema import BaseSchema
from marshmallow_dataclass import _internal_class_schema


class Outer:
    def __del__(self):
        print("deleting outer...")


@dataclass(frozen=True)
class Sample:
    a: str
    def __del__(self):
        print("deleting sample...")


def inner_demo():
    schema = marshmallow_dataclass.class_schema(Sample, base_schema=BaseSchema)()
    data = schema.load({"a": "a string"})
    return data


def demo():
    d = inner_demo()
    o = Outer()


for i in range(10):
    demo()
    print(_internal_class_schema.cache_info())

print("done with demo, no object deleted so far")
_internal_class_schema.cache_clear()
print("cleared cache")

No __del__ is called on either Sample or Outer during the method invocations, only after the cache_clear can the local variables be garbage collected. If you run the demo with 8.4.4 or just replace _internal_class_schema(clazz, base_schema, clazz_frame) with _internal_class_schema(clazz, base_schema, None) in marshmallow_dataclass.class_schema. Additionally this also demonstrates that the caching does no longer work as there no cache hits at all anymore.

[LostInDarkMath]

Hi,
I came across this issue because our software has a memory leak in production. After days of troubleshooting and narrowing down the problem, I came across this ticket.

Are there any plans to fix this in the near future?
And how can it be that such a critical bug remains unfixed for 2 years?

[lovasoa]

I understand your frustration, but please refrain from "how can it be", when referring to free and open source software. Instead, you can offer time or money to free software projects to unblock situations like this.

[dairiki]

This is (mostly) fixed by #258 (marshmallow_dataclass >= 8.6.1).

When necessary, references to frames are still held until the schema is constructed (by accessing the .Schema attribute). Memory leakage is still possible if large numbers of marshmallow_dataclasses are constructed without accessing their .Schema attribute.