Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"address" warnings about possible memory leaks due to lru_cache #73

Closed
yarikoptic opened this issue Jun 7, 2022 · 4 comments · Fixed by #74
Closed

"address" warnings about possible memory leaks due to lru_cache #73

yarikoptic opened this issue Jun 7, 2022 · 4 comments · Fixed by #74
Assignees

Comments

@yarikoptic
Copy link
Member

I have mentioned while looking at https://github.com/datalad/datalad-fuse/runs/6672682413?check_suite_focus=true of #72 that linting gives us

lint run-test: commands[0] | flake8 --config=tox.ini datalad_fuse
datalad_fuse/fsspec.py:53:6: B019 Use of `functools.lru_cache` or `functools.cache` on methods can lead to memory leaks. The cache may retain instance references, preventing garbage collection.
datalad_fuse/fsspec.py:206:6: B019 Use of `functools.lru_cache` or `functools.cache` on methods can lead to memory leaks. The cache may retain instance references, preventing garbage collection.
datalad_fuse/fuse_.py:110:6: B019 Use of `functools.lru_cache` or `functools.cache` on methods can lead to memory leaks. The cache may retain instance references, preventing garbage collection.

Needs analysis on either it is something to just ignore or we indeed would retain too many instances.

@jwodder
Copy link
Member

jwodder commented Jun 7, 2022

@yarikoptic Normally, datalad fusefs is meant to be run as a command, in which case all the instances of classes that use @lru_cache are deleted when the program exits anyway. I can only see this being a problem if someone were to invoke datalad.api.fusefs() multiple times in a single process.

Either way, the best option for replacing @lru_cache seems to be the methodtools package.

@yarikoptic
Copy link
Member Author

I can only see this being a problem if someone were to invoke datalad.api.fusefs() multiple times in a single process.

and even then I am not 100% certain it wouldn't be a "feature" -- don't we want to retain those instances as long as their methods are in the cache? I think we want, so let's just keep them and ignore those warnings for now.

Or am I missing some aspect?

@jwodder
Copy link
Member

jwodder commented Jun 8, 2022

@yarikoptic If datalad.api.fusefs() is called multiple times in the same process, even if it's passed the same arguments, new instances of the datalad-fusefs classes will be constructed, and they will not have access to cached results from previous instances.

@yarikoptic
Copy link
Member Author

ah -- gotcha. Let's then switch to methodtools

yarikoptic added a commit that referenced this issue Jun 8, 2022
Use methodtools.lru_cache
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants