[Support]: Frigate leaking GBs of memory #10921
-
Describe the problem you are havingFrigate has been leaking GBs of memory ever since upgrading from 0.12 to 0.13.2. I used to run on 4GB of RAM which Frigate fully consumed in less than 1h, I then bumped the memory of my instance to 8GB of RAM and Frigate still managed to eat it all overnight. Above shows a resident memory usage of the main frigate process at over 4.9GB which combined with all the other support processes was taking all 8GB of RAM. Doing a quick test over a 1h time period: So frigate seems to be leaking memory at a rate of just under 2MB/minute here. Version0.13.2-6476f8a Frigate config filemqtt:
host: PRIVATE
user: PRIVATE
password: PRIVATE
ffmpeg:
hwaccel_args: preset-vaapi
cameras:
shf-camera01:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera01
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 1920
height: 1080
objects:
track:
- cat
- person
record:
events:
objects: []
shf-camera02:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera02
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 1920
height: 1080
objects:
track:
- cat
- person
record:
events:
objects: []
shf-camera03:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera03
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 1920
height: 1080
objects:
track:
- cat
- person
record:
events:
objects: []
shf-camera04:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera04
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 1920
height: 1080
objects:
track:
- cat
- person
record:
events:
objects: []
shf-camera05:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera05
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2560
height: 1920
objects:
track:
- bird
record:
events:
objects: []
shf-camera06:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera06
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2560
height: 1920
objects:
track:
- cat
- person
snapshots:
enabled: True
shf-camera07:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera07
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2304
height: 1296
objects:
track:
- cat
- person
snapshots:
enabled: True
shf-camera08:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera08
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2560
height: 1920
objects:
track:
- car
- cat
- person
record:
events:
objects: []
shf-camera09:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera09
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2560
height: 1920
objects:
track:
- cat
- person
record:
events:
objects: []
shf-camera10:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera10
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2304
height: 1296
objects:
track:
- cat
- car
- person
snapshots:
enabled: True
shf-camera11:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera11
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2304
height: 1296
objects:
track:
- cat
- car
- person
snapshots:
enabled: True
required_zones:
- driveway
record:
events:
required_zones:
- driveway
zones:
driveway:
coordinates: 815,1019,2304,1088,2304,1296,0,1296,0,1168,483,943
shf-camera12:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera12
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2304
height: 1296
objects:
track:
- bird
- cat
- person
snapshots:
enabled: True
shf-camera13:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera13
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2304
height: 1296
objects:
track:
- cat
- car
- person
snapshots:
enabled: True
shf-camera14:
ffmpeg:
inputs:
- path: rtmp://PRIVATE:1935/shf-camera14
input_args: preset-rtmp-generic
roles:
- record
- detect
detect:
width: 2304
height: 1296
objects:
track:
- bird
- cat
- person
snapshots:
enabled: True
record:
enabled: True
retain:
days: 7
mode: all
events:
retain:
default: 10
mode: active_objects
post_capture: 0
pre_capture: 1
detectors:
coral:
type: edgetpu
device: usb
objects:
track: []
snapshots:
enabled: False
bounding_box: True
crop: true
retain:
default: 10
birdseye:
enabled: True
mode: continuous
live:
height: 720
quality: 5Relevant log outputThe process being completely out of memory resulted in no log entries at all for the affected time period (10h):
2024-04-09 05:38:48.358042876 2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:36 -0400] "GET /api/shf-camera11/latest.jpg?h=333 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 05:41:52.667999486 2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:53 -0400] "GET /api/shf-camera06/latest.jpg?h=444 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 05:41:52.668297252 2602:fc62:b:8005:216:3eff:fee4:cc60 - - [09/Apr/2024:01:27:53 -0400] "GET /api/shf-camera07/latest.jpg?h=333 HTTP/1.1" 499 0 "-" "HomeAssistant/2024.3.3 aiohttp/3.9.3 Python/3.11" "-"
2024-04-09 15:52:02.554579245 [INFO] Preparing Frigate...
2024-04-09 15:52:02.566412169 [INFO] Starting NGINX...
2024-04-09 15:52:02.663614853 [INFO] Preparing new go2rtc config...
2024-04-09 15:52:02.697756151 [INFO] Starting Frigate...
2024-04-09 15:52:03.570434492 [INFO] Starting go2rtc...
2024-04-09 15:52:03.682457180 11:52:03.682 INF go2rtc version 1.8.4 linux/amd64
2024-04-09 15:52:03.683346518 11:52:03.683 INF [api] listen addr=:1984
2024-04-09 15:52:03.683645144 11:52:03.683 INF [rtsp] listen addr=:8554
2024-04-09 15:52:03.684545352 11:52:03.684 INF [webrtc] listen addr=:8555FFprobe output from your cameraNot relevant as not camera specific.Frigate statsNo response Operating systemDebian Install methodDocker CLI Coral versionUSB Network connectionWired Camera make and modelMix of Reolink and bunch of other vendors (not relevant) Any other information that may be helpfulSeems similar to #10649 but in my case the system was just preventing memory allocations rather than triggering the kernel out of memory killer. |
Beta Was this translation helpful? Give feedback.
Replies: 9 comments 13 replies
-
|
A lot of time has been spent looking in to this. https://github.com/bloomberg/memray is a great tool for understanding where memory allocations are going. In my usage it appears there is no memory leak, the heap size does not increase at all, python just does not release the memory after it has been garbage collected. I don't think there is any simple / immediate solution. https://www.reddit.com/r/learnpython/comments/17j1aff/rss_memory_inflates_leaks_while_heap_memory_size/?rdt=33549 Perhaps in your case it is something different, a memray output would show what is using memory |
Beta Was this translation helpful? Give feedback.
-
|
Okay, the fact that there are reports of similar behavior in other python3 applications that similarly are running inside of resource constrained containers makes me want to look at a couple of options here, will report back in a few minutes (or hours if I need to do a full leak test). |
Beta Was this translation helpful? Give feedback.
-
|
Basically, the other reporter mentioned running this on a Proxmox server, so with Frigate running (with or without Docker, not sure) inside of a resource constrained LXC container. The container I usually give 4GB of RAM out of the 500GB+ of the host system. Now if python3 decided to be funny with caching and allow for memory to grow up till some arbitrary value determined from the total memory it think it can use, then that'd explain the problem. My first attempt at fixing this is simply by manually telling Docker to map LXCFS on /proc/meminfo, /proc/swaps, ... inside of the Frigate container. If python3 is looking at those files to figure out how much memory is available on the system, then that will work fine and it shouldn't get to a point where it runs the whole thing out of RAM anymore. If that doesn't work, then it's possible that python3 makes use of the |
Beta Was this translation helpful? Give feedback.
-
|
Still appears to be consuming memory at an alarming rate. I'm also tracing it for good measure and see that something within the python3 process does call the I can't tell what the information is used for but it certainly seems likely that this is the culprit and that I'll have to turn on interception to fix this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Right, initial test shows 8MB/minute when running under a 4GB total limit. I'll turn on the sysinfo interception and see if the behavior is different. |
Beta Was this translation helpful? Give feedback.
-
|
The every minute sys info call is most likely the stats which are presented via MQTT for the system page / integration |
Beta Was this translation helpful? Give feedback.
-
|
Ah, that'd make sense, though I guess the same underlying logic within python may be used for setting up some kind of memory limit on process startup. |
Beta Was this translation helpful? Give feedback.
-
|
And running with sysinfo interception, so far so good, process usage hovers around 485MB and seems to be actually releasing memory back to the OS following spikes. I'm still waiting for the results after keeping this going for an hour or so to make sure things are actually stable. If they are, then we'll know what python3 is doing now and how to fix it, at least for anyone using Incus, those not running on a container platform capable of system call emulation will be out of luck unfortunately. |
Beta Was this translation helpful? Give feedback.
-
|
just curious, what would that look like for users running in docker containers via podman? |
Beta Was this translation helpful? Give feedback.
-
|
Docker or Podman users will most likely run into the same issue if they specify a memory limit for their Frigate container as the python3 processes running inside of that container will not see the limit and therefore will plan their memory allocation based on the total amount of memory available on the system. Now I don't think that's necessarily a very common setup and even if it was, as Frigate is likely to get the lion's share of the system's memory anyways, the risk of python3 hitting the issue is pretty small. That's assuming that most Frigate users run on "normal" systems, Rpi4/Rpi5, Intel NUC and the like. In my case I'm running into the worst version of this because my server is a 48-core AMD EPYC with 500GB of RAM. So Frigate not seeing the limit of 4GB causes it to assume it's got 500GB to work with! |
Beta Was this translation helpful? Give feedback.
-
|
right, I am asking because I'd be curious to see how it behaves on my system which is an AMD 5700G Unraid NAS with 64GB of RAM but only 8GB allocated to frigate. I have never run out of memory but it does sometimes make its way up to 5GB or so |
Beta Was this translation helpful? Give feedback.
-
|
@andrewhack can you comment on how much memory your Proxmox host has? Your report shows you indeed have a 2GB cgroup limit on the container and that python3 consumed all of it causing the OOM killer to trigger, but the kernel trace doesn't show the total system memory you're working with. I'm kinda hoping that your system has 32GB+ of RAM. If it does, then it's basically a repeat of what I'm seeing, which is going to be annoying for you given you're on Proxmox and can't just turn on sysinfo emulation, but would at least be good news for anyone running Frigate on lower specced systems as they'd hopefully not hit the issue. |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 it really looks to me like python3 is aiming for a sweet spot of around 10% of system memory given I've now limited it to 4GB and it's staying below 500MB of RAM. So on your system with 64GB of RAM, I'd then expect the process to be in the 5-6GB range which would match up with what you're seeing. Now the issue is that if you were to double your system's RAM to 128GB, then Frigate would try to hit the 10-12GB RAM range and crash your container due to the 8GB limit. |
Beta Was this translation helpful? Give feedback.
-
|
Right, so a couple hours later and memory usage is stable, so the issue is confirmed to be related to python3 now calling |
Beta Was this translation helpful? Give feedback.
-
|
Finally someone found the reason! Thanks @stgraber, I have been having memory issues from the moment I upgraded to v0.13 and couldn´t figure out what was going on. Now that we know python3 is causing the issue by looking at the wrong place to know how much memory is available, is there a way to fix that? |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 I guess this issue must be happening to anyone running Frigate in a Proxmox LXC with a small percentage of memory assigned to the container in relation to the total memory available in the Proxmox server. My understanding is the only solution available at the moment is going back to Frigate v0.12 (please correct me if I am wrong), are you aware of any idea/plan to solve this issue in the future? |
Beta Was this translation helpful? Give feedback.
-
|
frigate version has no effect on this issue. If you are not seeing it in frigate 0.12 but are seeing it in frigate 0.13 then you are seeing a different issue or a combination of issues |
Beta Was this translation helpful? Give feedback.
-
|
@NickM-27 as @stgraber mentioned in the description of this defect, in my case I also started having this memory issue after upgrading from Frigate v0.12 to v0.13, so I assumed python3 was being used in a different way in v0.12. But regarthless of the version this started, this seems like a really important issue affecting a lot of Frigate users, and we found the reason for the issue in this thread which is great, now the next question is can it be fixed? |
Beta Was this translation helpful? Give feedback.
-
|
hello ok it's a problem of python3 and not frigate but we can't do anything ? Usage of RAM is really a problem |
Beta Was this translation helpful? Give feedback.
-
|
I reboot the LXC every 3 days to save RAM and SWAP :'(
|
Beta Was this translation helpful? Give feedback.
-
|
What I do is limitting the maximum RAM assigned to the container in the docker-compose.yaml file using the parameter mem_limit, so when Frigate tries to use more memory it just crashes and restart, but these solutions are just workarounds that don´t fix the underlying issue. We need to find a way for python3 to be aware of the real amount of available memory so it doesn´t use so much memory. |
Beta Was this translation helpful? Give feedback.
-
|
weird. limit memory does not work for me total : 8 Gib // limited of the container : 4 Gib stay a 1 Gib / 1.5 Gib but memory usage increases |
Beta Was this translation helpful? Give feedback.
Right, so a couple hours later and memory usage is stable, so the issue is confirmed to be related to python3 now calling
sysinfowhich if that returns an amount of total memory higher than what's allowed for the container will cause python3 to start growing the process memory until it crashes.