-
Notifications
You must be signed in to change notification settings - Fork 221
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
High memory consumption for long-running tasks #169
Comments
Yes, this is not too surprising, as pyinstrument is recording frame samples as the program executes. If you need to reduce the memory consumption, for a long-running job, you might try reducing the interval a bit. The default is to record every millisecond, but this might be too often for a long running job. |
Thanks for the response.
means that the code will be running slower with smaller interval value, not that memory will be consumed like crazy. In addition, as I understand, it's not possible to set the interval value from CLI. Plus, it's not a solution, more like a workaround. Memory consumption still increases only now with a slower pace (with bigger interval value). |
This would be great. We have some jobs which can take 20 minutes (or longer), and by that point the memory overhead of pyinstrument grows to >20gb. Without pyinstrument it's under 5gb. On some systems this causes out of memory errors. |
Thanks for chiming in. I'm curious if reducing the profiling interval would be a solution for your use case, or if you really need millisecond accuracy. The issue I see with dumping intermediate output to disk is that at some point pyinstrument is going to have to load it all into RAM to analyse it and produce reports, which is going to hit the same RAM limits as you're seeing during execution. |
I don't suppose there is any way to compress or filter out unnecessary data during the runtime? |
Compress, maybe, but we'd still have the issue when analysing. What would you consider unnecessary data? |
I am using pyinstrument for profiling some ML/NLU specific tasks and noticed that when running profiling on function with a lot of simple computations inside memory consumption stedealy increases.
For example such a dummy code:
when running
python main.py
consumes ~3.2 MB, whilepyinstrument main.py
- ~40 MB.And the main issue is that while running without profiling memory consumption stays at the same level, with profiling - constantly increases.
The text was updated successfully, but these errors were encountered: