-
-
Notifications
You must be signed in to change notification settings - Fork 786
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Django Channels Memory Leak on every message or connection #2094
Comments
Does the same thing happen with other protocol servers, such as hypercorn and Daphne? |
I've tested The interesting thing is, while Here are the commands I used for each:
|
And can you see from any of the tools, memray perhaps, which objects are consuming the memory? (I'd expect a |
@cacosandon Also, can you try with the PubSub layer, and see if the results are different there? Thanks. |
Sure! I'll try to find time today to prepare a report on |
So I tried multiple combinations. All HTML reports from But below there are screenshots from them. First, tried with Redis Channels (not PubSub) to get memory leaks. With
So, the leaks report include memory that was never released back, but I don't know how to interpret it correctly. Seems like
Here is the screenshot of the
Then tried with
The interesting part is that
Then, I tried with garbage collect for
And finally tried with
Just in case, I also removed all Hope all these reports help understanding the constant memory increase. Right now I am trying to move my application to |
I've made it to make For some reason the websocket messages that were bytes-only were sent as Added a PR for that: #2097 async def websocket_receive(self, message):
"""
Called when a WebSocket frame is received. Decodes it and passes it
to receive().
"""
- if "text" in message:
+ if "text" in message and message["text"] is not None:
await self.receive(text_data=message["text"])
else:
await self.receive(bytes_data=message["bytes"]) Testing now in staging 🤞 |
Hi @cacosandon Looking at the uploaded report, for e.g. The |
Hey @carltongibson, thank you for taking a look. Yep, but if you zoom in On the other hand, |
@carltongibson, do you have any clue about what's happening? Or what else can I try? I'm willing to try anything! |
@cacosandon Given that you report it happening with the pub sub layer and different servers, not really. You need to identify where the leak is happening. Then it's possible to say something. |
@carltongibson all my samples are from using Some things I've noticed:
I don't know how nobody else is having this problem. Maybe they just don't send large messages 🤔 |
Hi @cacosandon — are you able to identify where the leak is happening? Alas, I haven't had time to dig into this further for you. Without that it's difficult to say too much. If you can identify a more concrete issue, there's a good chance we can resolve it. |
@carltongibson no :( that's actually the thing that I'm struggling on: finding the memory leak 😓 I really tried every tool to detect it, but nothing noticeable or strange in the reports.. |
I wouldn't assume that. 😉 I've been silently watching and hoping you find more than I did when I looked. We had some success changing servers from Here are some other things I've watched: |
@mitgr81 what tools do you use to monitor and restart? For now I would love to implement that. Will take a look on those resources! |
We're rocking a bespoke monitor for docker containers. It's pretty simple; essentially we label each container with a valid restart time and a memory limit (among other rules); and the "container keeper" looks for them. |
I'm having a memory leak in Django Channels using
uvicorn
.Every "memory crash" is a restart/deploy 👇
This not just happens within my project, but also with the tutorial basic chat example.
Here is the repository with that minimal example and memory profiling: https://github.com/cacosandon/django-channels-memory-leak
This happens locally, in the server, with/without
DEBUG
, just by reconnecting or sending messages (in the example I've added large messages so you can notice the memory leak).The memory is never released.. even if the user disconnects after.
I've proved it with
memory-profiler
andmemray
(both commands were added in the README so you can reproduce)Dependencies:
I (think that) have really tried everything; deleting objects, manual garbage collection, etc. Nothing prevents the memory to increase and to never be released back. Any insights? 🙏
The text was updated successfully, but these errors were encountered: