Frigate 0.13 RC: No frames have been received, check error logs #9406
Comments
|
Have you recompiled the trt models to use the new version? Like the release notes said models generated previously are not supported. |
No, I'm not using TRT Models here, it's just the homonymous name of folder where I placed TRT & TFLITE models. |
|
The main change in 0.13 is that models are generated automatically for trt on first startup. What are the frigate logs after running for 5 minutes? |
|
The Firgate logs are unchanged after running for 5 minutes: Was something changed in FFMPEG interaction in 0.13? I see frames skipped in Detect here, which was not the issue in 0.12 (0 skipped there all the time): The weird thing is that camera screenshots on the main Frigate dashboard are not getting updated after Frigate restart. And the more I refresh the page browser the less screenshots I see (I see "No frames have been received, check error logs" messages on all camera preview windows). In 0.12 I saw regular camera image updates (once per 5 seconds or so). |
|
No nothing changed with ffmpeg. Also if I understand what you're saying that is not correct, there is no such functionality to refresh the images on the main camera dashboard. Can you send a full screenshot of the entire system page please |
|
@NickM-27 Yes, you're right, I misinterpreted it a bit: after I refresh the page in 0.12 I see changes in camera images, while in 0.13 I see either a static (previous) image or the message about "No frames.." on a black screen. |
|
100 milliseconds of inference time is crazy high, so that is most likely the problem. I'd suggest using the built in model and see if that fixes the issue |
Same inference time in 0.12, and no problems at all, not a single frame skipped. Built-in model, albeit gives lesser inference time, quality-wise is much more inferior to EfficientDet. There's something wrong/different in 0.13 for sure. |
|
The skipped frames counter was broken in 0.12, so no skipped frames doesn't mean it wasn't actually skipping frames |
|
100ms is really slow, it means you can only run 10 inferences per second, which is not nearly enough for 5 fps across 6 cameras |
|
In any case, we've not had any other reports of this issue so it will likely be something fairly specific |
|
@NickM-27 OK, I can try reconfiguring Frigate for TensorRT despite its model's inferiority in terms of accuracy, but I don't think the inference time has something to do with constantly disappearing streams in 0.13. When "No frames.." appears, the system doesn't work, logically. And in 0.12 it works. |
|
I've tried running with TRT, now the inference time is much better, but it doesn't change the "No frames.." issue appearing periodically (very frequently). |
|
the inference time has a direct affect on this issue, because when frigate is sitting waiting for the object detection inference to complete it is unable to process other frames and that means the frame times fall behind causing frigate to return the no frames received image. we still see loads of skipped frames which means something is still very wrong here. I wouldn't expect that to occur with 47 ms inference speed but it seems something is quite wrong here. I think the best thing for testing is to use the built in coral model and that should give an inference time of 10 ms and see if it still occurs at that point |
|
another thing you could try is just turning off detect in the dashboard for all cameras and see if the issue improves |
|
@NickM-27 Yeah, I managed to get 27-33 ms inference time with TRT Yolov7-640 model. However I regret that the EfficientDet model takes so much time for inference. That model is much more false-proof than the standard one. |
|
Because the models aren't trained on security camera images they definitely vary highly from camera to camera in terms of false positives. |
|
@NickM-27 One more question - I have a PTZ camera with parameters like this: In Frigate 0.12 there was a clear recording from motion start till motion end (PTZ moves quickly, I assume the motion detection was leveraged immediately and recording started as well). |
|
there are some things to clear up here:
|
|
I have the same problem, and in version 0.12 it doesn't happen, only in version 13
If anyone has any help, I have Reolink cameras. but in version 12 it works |
|
I have Reolink RLC-523WA actually two separate HA setups. With Frigate 0.12.1 both setups are working, but upgrading Frigate to 0.13.2 those are not working. By restoring back to Frigate 0.12.1 without any Config changes, the setups are working. |
|
Very similar here. I get the 'No frames have been received, check error logs' very often on all cameras simultaneously. This wasn't happening on previous versions. |
I have this nowadays working. It seems that it takes time to stabilize after upgrade from 0.12.1 to 0.13.2. It was taking even about 24 Hours. |
|
I also have this same issue with a variety of Amcrest cameras after upgrading to 0.13.2. Nearly all of my cameras are giving the "no frames have been received" message, with a few intermittently successful frames being shown after multiple reloads of the camera page. Everything worked flawlessly with 0.12.1, so I've had to downgrade back to that version to have a functional NVR system. |
|
So, after some fiddling, I found that I was able to get 0.13.2 working with a combination of using the ffmpeg:rtsp:// source in go2rtc along with the secondary lower VGA resolution camera streams at 640x480, then scaled up to 854x480 for detection to fix the aspect ratio. Previously with 0.12.1 I had been using the native camera resolution video scaled down to 1280x720 for detection without problem. I'm still somewhat disappointed in this workaround, as JSMPEG quality suffers quite dramatically, but at least my home assistant integration is working again with the latest version. |
|
Also getting this after moving to v0.13. Though I am guessing it is due to performance and high inference. |
|
I'm getting this too after upgrading to v0.13. I'm having reolink camera with the recommended setup (using HTTP stream with lower resolution). I haven't found a solution yet. Part of the config (worked really well with v0.12) Logs |
|
Update: Downgrading go2rtc to 1.6.2 seems to fix the issue. I tested 1.7.0 and got the same issue. |
|
@tobernguyen the config you have there is not the reolink recommended config, the difference being the recommended config uses ffmpeg:http |
|
I couldn't solve the problem, I had to downgrade to 12. I would have to try changing the go2rtc version. Someone who confirmed if that would be the problem? |
@NickM-27 Thanks for pointing that out. I will consider trying that with the latest go2rtc. That said, I have used this configuration in v0.12 and have zero issues. Since downgrading go2rtc to 1.6.2, I have never seen the "No frames have been received, check error logs" anymore and the recordings are no longer broken midway. @ithesk At least in my setup, I can confirm go2rtc was the culprit ^ |
|
@tobernguyen yes, the later go2rtc has issues with http flv hence the recommendation to use ffmpeg with it |
I just tested using |
|
I changed it to |
how to download the version of go2rtc? |
|
@jaaneo try following the tips on this thread #6502 (comment). Hopefully it steers you in the right path as all installations are slightly different. |
|
Hi, I also just wanted to add that I am having issues with the OP's topic. Tried to change go2rtc version from 1.8.4 to 1.8.5 no change, to 1.6.0 no change. to 1.9.0 no change. I can see how the streamy are being picked up by the GPU but immediately disapear from nvidia-smi. Only one out of six camera (all of my six cams are of the same model [Annke c800; HVEC only]) is working without any issues. Storage or GPU Memory (tried 4GB to 24GB on a Tesla P40) cannot be an issue... since streams are aborting after only (max) 5 seconds. If needed I can attache some logs. Best, |
just go here and download what you need: https://github.com/AlexxIT/go2rtc/releases |
|
@Goeste I tried all those versions you did as well and none of them worked for me. Go2rtc version 1.5.0 however has been perfect, super stable no issues across all my POE and WiFi based cameras. Give 1.5.0 a go and good luck 🤞 |
|
@sebdoan TY for that ;) guess what.. i rebootet all cams that were not working and now even with 1.9.0 everything works again... let's see for how long if it will fail again, i'll switch to 1.5.0 |
|
Ok, with 1.9.0. it failed again. I replade go2rtc with version 1.5.0 still failing... only one webcam is detecting objects, non of the other 5 is doing so... |
|
I also see it's only failing when using the mainstream instead of the substream for detection (except for one camera....) weird... |
|
OK, issue found. all webcams were set to 6FPS where as the one that is working was on 12,5FPS. So... I set all others to 12,5 FPS and automagically streams reappear... no hanging since with go2rtc 1.9.1 :) My webcams are all Annke c800 (h265 streams) |
Describe the problem you are having
No frames have been received, check error logs appears periodically in Frigate's Web Interface
Restart of Frigate may temporarily help with some cameras, but soon the problem reappears.
After a while camera image may show up again normally.
There were no problems like this with the previously used Frigate 0.12.1 Release.
I tried to use the embedded go2rtc 1.8.4, then after no lock included a go2rtc 1.8.5 in docker volume, but nothing's changed.
Docker-compose.yml:
Version
0.13 RC1
Frigate config file
Relevant log output
Frigate stats
{"cameras":{"back":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":495,"detection_enabled":1,"detection_fps":3.1,"ffmpeg_pid":516,"pid":475,"process_fps":3.1,"skipped_fps":2.1},"front":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":490,"detection_enabled":1,"detection_fps":3.0,"ffmpeg_pid":512,"pid":473,"process_fps":0.5,"skipped_fps":4.6},"ptz":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":486,"detection_enabled":1,"detection_fps":3.0,"ffmpeg_pid":491,"pid":472,"process_fps":1.5,"skipped_fps":3.5},"sosed":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.1,"capture_pid":515,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":521,"pid":477,"process_fps":5.1,"skipped_fps":0.0},"trucks":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.0,"capture_pid":520,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":528,"pid":479,"process_fps":5.0,"skipped_fps":0.0},"trucks_back":{"audio_dBFS":0.0,"audio_rms":0.0,"camera_fps":5.0,"capture_pid":525,"detection_enabled":1,"detection_fps":0.0,"ffmpeg_pid":531,"pid":483,"process_fps":5.0,"skipped_fps":0.0}},"cpu_usages":{"1":{"cmdline":"/package/admin/s6/command/s6-svscan -d4 -- /run/service","cpu":"0.0","cpu_average":"0","mem":"0.0"},"15":{"cmdline":"s6-supervise s6-linux-init-shutdownd","cpu":"0.0","cpu_average":"0","mem":"0.0"},"16":{"cmdline":"/package/admin/s6-linux-init/command/s6-linux-init-shutdownd -c /run/s6/basedir -g 3000 -C -B","cpu":"0.0","cpu_average":"0","mem":"0.0"},"24":{"cmdline":"s6-supervise s6rc-fdholder","cpu":"0.0","cpu_average":"0","mem":"0.0"},"25":{"cmdline":"s6-supervise s6rc-oneshot-runner","cpu":"0.0","cpu_average":"0","mem":"0.0"},"26":{"cmdline":"s6-supervise nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"27":{"cmdline":"s6-supervise go2rtc-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"28":{"cmdline":"s6-supervise go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"29":{"cmdline":"s6-supervise frigate-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"30":{"cmdline":"s6-supervise nginx-log","cpu":"0.0","cpu_average":"0","mem":"0.0"},"31":{"cmdline":"s6-supervise frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"32":{"cmdline":"s6-supervise go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"43":{"cmdline":"/package/admin/s6-2.11.3.2/command/s6-fdholderd -1 -i data/rules","cpu":"0.0","cpu_average":"0","mem":"0.0"},"44":{"cmdline":"/package/admin/s6/command/s6-ipcserverd -1 -- /package/admin/s6/command/s6-ipcserver-access -v0 -E -l0 -i data/rules -- /package/admin/s6/command/s6-sudod -t 30000 -- /package/admin/s6-rc/command/s6-rc-oneshot-run -l ../.. --","cpu":"0.0","cpu_average":"0","mem":"0.0"},"89":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"90":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/go2rtc","cpu":"0.0","cpu_average":"0","mem":"0.0"},"91":{"cmdline":"s6-log -b -- T 1 n0 s10000000 T /dev/shm/logs/frigate","cpu":"0.0","cpu_average":"0","mem":"0.0"},"98":{"cmdline":"/config/go2rtc -config=/dev/shm/go2rtc.yaml","cpu":"0.0","cpu_average":"0","mem":"0.0"},"106":{"cmdline":"bash ./run.user go2rtc-healthcheck","cpu":"0.0","cpu_average":"0","mem":"0.0"},"107":{"cmdline":"python3 -u -m frigate","cpu":"4.4","cpu_average":"3","mem":"1.4"},"121":{"cmdline":"nginx: master process nginx","cpu":"0.0","cpu_average":"0","mem":"0.0"},"154":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"155":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"156":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"157":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"159":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"172":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"192":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"225":{"cmdline":"nginx: worker process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"266":{"cmdline":"nginx: cache manager process","cpu":"0.0","cpu_average":"0","mem":"0.0"},"448":{"cmdline":"frigate.logger ","cpu":"0.0","cpu_average":"0","mem":"0.3"},"450":{"cmdline":"frigate.recording_manager","cpu":"0.0","cpu_average":"2","mem":"0.4"},"459":{"cmdline":"/usr/bin/python3 -c from multiprocessing.resource_tracker import main;main(61)","cpu":"0.5","cpu_average":"0","mem":"0.0"},"460":{"cmdline":"frigate.detector.coral","cpu":"76.8","cpu_average":"69","mem":"0.5"},"462":{"cmdline":"frigate.output ","cpu":"2.4","cpu_average":"0","mem":"0.3"},"468":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1920x1080 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"469":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 3840x2160 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"471":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 3840x2160 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"472":{"cmdline":"frigate.process:ptz ","cpu":"3.9","cpu_average":"2","mem":"0.4"},"473":{"cmdline":"frigate.process:front","cpu":"4.4","cpu_average":"6","mem":"0.4"},"475":{"cmdline":"frigate.process:back ","cpu":"4.4","cpu_average":"3","mem":"0.4"},"476":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2688x1520 -i pipe: -f mpegts -s 1273x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"477":{"cmdline":"frigate.process:sosed","cpu":"1.0","cpu_average":"2","mem":"0.4"},"478":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2688x1520 -i pipe: -f mpegts -s 1273x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"479":{"cmdline":"frigate.process:trucks","cpu":"1.5","cpu_average":"3","mem":"0.4"},"483":{"cmdline":"frigate.process:trucks_back","cpu":"1.0","cpu_average":"3","mem":"0.4"},"484":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 2592x1944 -i pipe: -f mpegts -s 960x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"486":{"cmdline":"frigate.capture:ptz ","cpu":"3.9","cpu_average":"2","mem":"0.3"},"489":{"cmdline":"ffmpeg -f rawvideo -pix_fmt yuv420p -video_size 1280x720 -i pipe: -f mpegts -s 1280x720 -codec:v mpeg1video -q 8 -bf 0 pipe:","cpu":"0.0","cpu_average":"0","mem":"0.0"},"490":{"cmdline":"frigate.capture:front","cpu":"8.8","cpu_average":"8","mem":"0.4"},"491":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.8:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/ptz@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale=1920:1080 -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"31.6","cpu_average":"33","mem":"0.1"},"495":{"cmdline":"frigate.capture:back ","cpu":"10.2","cpu_average":"8","mem":"0.4"},"512":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.4:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/front@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"19.0","cpu_average":"19","mem":"0.5"},"515":{"cmdline":"frigate.capture:sosed","cpu":"3.9","cpu_average":"4","mem":"0.4"},"516":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.5:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/back@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=3840:h=2160:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"18.5","cpu_average":"17","mem":"0.4"},"520":{"cmdline":"frigate.capture:trucks","cpu":"5.3","cpu_average":"4","mem":"0.4"},"521":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.6:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/sosed@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=2688:h=1520:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"11.2","cpu_average":"10","mem":"0.3"},"525":{"cmdline":"frigate.capture:trucks_back","cpu":"6.3","cpu_average":"5","mem":"0.4"},"528":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.7:554/axis-media/media.amp -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/trucks@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=2688:h=1520:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"11.7","cpu_average":"9","mem":"0.3"},"531":{"cmdline":"ffmpeg -hide_banner -loglevel warning -threads 2 -hwaccel cuda -hwaccel_output_format cuda -user_agent FFmpeg Frigate/0.13.0-49814b3 -avoid_negative_ts make_zero -fflags +genpts+discardcorrupt -rtsp_transport tcp -timeout 5000000 -use_wallclock_as_timestamps 1 -i rtsp://admin:pass@192.168.0.13:554/trackID=1 -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c copy -an /tmp/cache/trucks_back@%Y%m%d%H%M%S%z.mp4 -r 5 -vf fps=5,scale_cuda=w=2592:h=1944:format=nv12,hwdownload,format=nv12,format=yuv420p -threads 2 -f rawvideo -pix_fmt yuv420p pipe:","cpu":"13.6","cpu_average":"12","mem":"0.4"},"3228":{"cmdline":"sleep 30s","cpu":"0.0","cpu_average":"0","mem":"0.0"}},"detection_fps":9.1,"detectors":{"coral":{"detection_start":1705858405.229372,"inference_speed":114.51,"pid":460}},"gpu_usages":{"NVIDIA GeForce GTX 1650":{"dec":"45.0%","enc":"0.0%","gpu":"4.0%","mem":"33.61%"}},"processes":{"go2rtc":{"pid":98},"logger":{"pid":448},"recording":{"pid":450}},"service":{"last_updated":1705858405,"latest_version":"0.12.1","storage":{"/dev/shm":{"free":376.8,"mount_type":"tmpfs","total":512.0,"used":135.2},"/media/frigate/clips":{"free":1617886.5,"mount_type":"fuseblk","total":3815430.0,"used":2197543.4},"/media/frigate/recordings":{"free":1617886.5,"mount_type":"fuseblk","total":3815430.0,"used":2197543.4},"/tmp/cache":{"free":912.1,"mount_type":"tmpfs","total":953.7,"used":41.6}},"temperatures":{},"uptime":432,"version":"0.13.0-49814b3"}}Operating system
Debian
Install method
Docker Compose
Coral version
USB
Any other information that may be helpful
Go2RTC logs contain no errors:
The text was updated successfully, but these errors were encountered: