[Detector Support]: Coral hangs/crashes when using Frigate+ model #11315
-
Describe the problem you are havingI am currently using only one camera with Frigate, and it has been consistently working (USB Coral). Version0.13.2-6476F8A Frigate config filemqtt:
enabled: true
host: 172.17.0.1
user: mosquitto
password: xxxxxxxxxx
detectors:
coral:
type: edgetpu
device: usb
objects:
track:
- car
- person
- face
- license plate
- deer
- ups
- fedex
- package
- dog
- cat
- bird
model:
path: plus:<edgetpu model from frigate+>
ffmpeg:
hwaccel_args: preset-vaapi
record:
enabled: true
retain:
days: 0
mode: all
events:
retain:
default: 30
mode: motion
snapshots:
enabled: true
retain:
default: 30
cameras:
front_yard:
enabled: true
detect:
width: 1920
height: 1080
ffmpeg:
inputs:
- path: rtsp://xxxxxxxx:zzzzzzzzz@192.168.1.109/live0
roles:
- detect
motion:
mask:
- 1350,1080,1500,850,1583,705,1662,529,1719,324,1775,0,1920,0,1920,1080
- 577,1008,469,838,328,913,313,903,200,666,237,606,159,307,124,0,0,0,0,1080,447,1080
- 898,88,1117,140,1260,183,1449,229,1765,350,1814,0,128,0,137,261
zones:
own_property:
coordinates: 1920,1080,1920,441,1314,222,956,124,0,341,0,1080
record:
events:
required_zones: [own_property]
snapshots:
required_zones: [own_property]
test:
enabled: false
ffmpeg:
inputs:
- path: rtsp://xxxxxxxxxxxx:8554/zzzzzzz
roles:
- detectdocker-compose file or Docker CLI commandservices:
frigate:
privileged: true
container_name: Frigate
hostname: frigate
image: ghcr.io/blakeblackshear/frigate:stable
shm_size: 1024mb
volumes:
- ./config:/config
- /mnt/backups/nvr:/media/frigate
- type: tmpfs
target: /tmp/cache
tmpfs:
size: 1000000000
devices:
- /dev/bus/usb:/dev/bus/usb
- /dev/dri/renderD128:/dev/dri/renderD128
- /dev/dri/card0:/dev/dri/card0
environment:
TZ: America/Montreal
FRIGATE_RTSP_PASSWORD: zzzzzzzzzzzzz
LIBVA_DRIVER_NAME: radeonsi
PLUS_API_KEY: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
restart: always
ports:
- 8555:8555/tcp # WebRTC over tcp
- 8555:8555/udp # WebRTC over udp
labels:
traefik.enable: true
traefik.http.routers.frigate.entrypoints: web,websecure
traefik.http.routers.frigate.rule: Host(`xxxxxxxxxxxxxx.domain.com`)
traefik.http.services.frigate.loadbalancer.server.port: 5000
traefik.http.routers.frigate.tls: true
traefik.http.routers.frigate.tls.certresolver: production
traefik.http.middlewares.frigate.ipwhitelist.sourcerange: 127.0.0.1, 192.168.1.0/22
traefik.http.routers.frigate.middlewares: frigate
networks:
default:
name: it_traefik
external: trueRelevant log outputFrigate | s6-rc: info: service s6rc-fdholder: starting
Frigate | s6-rc: info: service s6rc-oneshot-runner: starting
Frigate | s6-rc: info: service s6rc-oneshot-runner successfully started
Frigate | s6-rc: info: service fix-attrs: starting
Frigate | s6-rc: info: service s6rc-fdholder successfully started
Frigate | s6-rc: info: service fix-attrs successfully started
Frigate | s6-rc: info: service legacy-cont-init: starting
Frigate | s6-rc: info: service legacy-cont-init successfully started
Frigate | s6-rc: info: service log-prepare: starting
Frigate | s6-rc: info: service log-prepare successfully started
Frigate | s6-rc: info: service nginx-log: starting
Frigate | s6-rc: info: service go2rtc-log: starting
Frigate | s6-rc: info: service frigate-log: starting
Frigate | s6-rc: info: service nginx-log successfully started
Frigate | s6-rc: info: service go2rtc-log successfully started
Frigate | s6-rc: info: service go2rtc: starting
Frigate | s6-rc: info: service go2rtc successfully started
Frigate | s6-rc: info: service go2rtc-healthcheck: starting
Frigate | s6-rc: info: service frigate-log successfully started
Frigate | s6-rc: info: service frigate: starting
Frigate | s6-rc: info: service go2rtc-healthcheck successfully started
Frigate | s6-rc: info: service frigate successfully started
Frigate | s6-rc: info: service nginx: starting
Frigate | 2024-05-09 18:01:09.507058210 [INFO] Preparing new go2rtc config...
Frigate | s6-rc: info: service nginx successfully started
Frigate | s6-rc: info: service legacy-services: starting
Frigate | 2024-05-09 18:01:09.515532365 [INFO] Preparing Frigate...
Frigate | 2024-05-09 18:01:09.519728780 [INFO] Starting NGINX...
Frigate | 2024-05-09 18:01:09.556755606 [INFO] Starting Frigate...
Frigate | s6-rc: info: service legacy-services successfully started
Frigate | 2024-05-09 18:01:10.172836393 [INFO] Starting go2rtc...
Frigate | 2024-05-09 18:01:10.292244850 14:01:10.292 INF go2rtc version 1.8.4 linux/amd64
Frigate | 2024-05-09 18:01:10.293801098 14:01:10.293 INF [rtsp] listen addr=:8554
Frigate | 2024-05-09 18:01:10.294441362 14:01:10.294 INF [api] listen addr=:1984
Frigate | 2024-05-09 18:01:10.297528336 14:01:10.297 INF [webrtc] listen addr=:8555
Frigate | 2024-05-09 18:01:12.310186614 [2024-05-09 14:01:12] frigate.app INFO : Starting Frigate (0.13.2-6476f8a)
Frigate | 2024-05-09 18:01:12.428281604 [tcp @ 0x21fa280] Connection to tcp://xxxxxxxxxxxxx:8554?timeout=0 failed: Connection refused
Frigate | 2024-05-09 18:01:12.429839889 [ERROR:0@1.209] global cap.cpp:164 open VIDEOIO(CV_IMAGES): raised OpenCV exception:
Frigate | 2024-05-09 18:01:12.429848593
Frigate | 2024-05-09 18:01:12.429862269 OpenCV(4.7.0) /io/opencv/modules/videoio/src/cap_images.cpp:253: error: (-5:Bad argument) CAP_IMAGES: can't find starting number (in the name of file): rtsp://xxxxxxxxxxxxxxxxx:8554/zzzzzzz in function 'icvExtractPattern'
Frigate | 2024-05-09 18:01:12.429867793
Frigate | 2024-05-09 18:01:12.429870564
Frigate | 2024-05-09 18:01:12.485604523 [2024-05-09 14:01:12] peewee_migrate.logs INFO : Starting migrations
Frigate | 2024-05-09 18:01:12.493829389 [2024-05-09 14:01:12] peewee_migrate.logs INFO : There is nothing to migrate
Frigate | 2024-05-09 18:01:12.507109566 [2024-05-09 14:01:12] frigate.app INFO : Recording process started: 227
Frigate | 2024-05-09 18:01:12.514476662 [2024-05-09 14:01:12] frigate.app INFO : go2rtc process pid: 88
Frigate | 2024-05-09 18:01:12.584243100 [2024-05-09 14:01:12] frigate.app INFO : Output process started: 242
Frigate | 2024-05-09 18:01:12.629336489 [2024-05-09 14:01:12] frigate.app INFO : Camera processor started for front_yard: 248
Frigate | 2024-05-09 18:01:12.629731447 [2024-05-09 14:01:12] frigate.app INFO : Camera processor not started for disabled camera test
Frigate | 2024-05-09 18:01:12.657896763 [2024-05-09 14:01:12] frigate.app INFO : Capture process started for front_yard: 250
Frigate | 2024-05-09 18:01:12.658278048 [2024-05-09 14:01:12] frigate.app INFO : Capture process not started for disabled camera test
Frigate | 2024-05-09 18:01:15.405151287 [2024-05-09 14:01:12] detector.coral INFO : Starting detection process: 237
Frigate | 2024-05-09 18:01:15.405158232 [2024-05-09 14:01:12] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb
Frigate | 2024-05-09 18:01:15.412242577 [2024-05-09 14:01:15] frigate.detectors.plugins.edgetpu_tfl INFO : TPU found
Frigate | 2024-05-09 18:01:19.503273515 [INFO] Starting go2rtc healthcheck service...
Frigate | 2024-05-09 18:01:32.996755638 [2024-05-09 14:01:32] frigate.watchdog INFO : Detection appears to be stuck. Restarting detection process...
Frigate | 2024-05-09 18:01:32.996761483 [2024-05-09 14:01:32] root INFO : Waiting for detection process to exit gracefully...
Frigate | 2024-05-09 18:02:03.014230614 [2024-05-09 14:02:03] root INFO : Detection process didnt exit. Force killing...
Frigate | 2024-05-09 18:02:03.228782089 [2024-05-09 14:02:03] root INFO : Detection process has exited...
Frigate | 2024-05-09 18:02:03.272035085 [2024-05-09 14:02:03] detector.coral INFO : Starting detection process: 353
Frigate | 2024-05-09 18:02:29.643250456 [2024-05-09 14:02:03] frigate.detectors.plugins.edgetpu_tfl INFO : Attempting to load TPU as usb
Frigate | 2024-05-09 18:02:29.644326874 Process detector:coral:
Frigate | 2024-05-09 18:02:29.644334659 [2024-05-09 14:02:29] frigate.detectors.plugins.edgetpu_tfl ERROR : No EdgeTPU was detected. If you do not have a Coral device yet, you must configure CPU detectors.
Frigate | 2024-05-09 18:02:29.645501969 Traceback (most recent call last):
Frigate | 2024-05-09 18:02:29.645533240 File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 160, in load_delegate
Frigate | 2024-05-09 18:02:29.645535193 delegate = Delegate(library, options)
Frigate | 2024-05-09 18:02:29.645537025 File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 119, in __init__
Frigate | 2024-05-09 18:02:29.645538429 raise ValueError(capture.message)
Frigate | 2024-05-09 18:02:29.645600404 ValueError
Frigate | 2024-05-09 18:02:29.645643940
Frigate | 2024-05-09 18:02:29.645646015 During handling of the above exception, another exception occurred:
Frigate | 2024-05-09 18:02:29.645647411
Frigate | 2024-05-09 18:02:29.645685474 Traceback (most recent call last):
Frigate | 2024-05-09 18:02:29.646175312 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
Frigate | 2024-05-09 18:02:29.646180254 self.run()
Frigate | 2024-05-09 18:02:29.646181878 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
Frigate | 2024-05-09 18:02:29.646183317 self._target(*self._args, **self._kwargs)
Frigate | 2024-05-09 18:02:29.646184906 File "/opt/frigate/frigate/object_detection.py", line 102, in run_detector
Frigate | 2024-05-09 18:02:29.646186371 object_detector = LocalObjectDetector(detector_config=detector_config)
Frigate | 2024-05-09 18:02:29.646189245 File "/opt/frigate/frigate/object_detection.py", line 53, in __init__
Frigate | 2024-05-09 18:02:29.646194115 self.detect_api = create_detector(detector_config)
Frigate | 2024-05-09 18:02:29.646195980 File "/opt/frigate/frigate/detectors/__init__.py", line 18, in create_detector
Frigate | 2024-05-09 18:02:29.646197421 return api(detector_config)
Frigate | 2024-05-09 18:02:29.646198987 File "/opt/frigate/frigate/detectors/plugins/edgetpu_tfl.py", line 41, in __init__
Frigate | 2024-05-09 18:02:29.646234482 edge_tpu_delegate = load_delegate("libedgetpu.so.1.0", device_config)
Frigate | 2024-05-09 18:02:29.646236519 File "/usr/lib/python3/dist-packages/tflite_runtime/interpreter.py", line 162, in load_delegate
Frigate | 2024-05-09 18:02:29.646249267 raise ValueError('Failed to load delegate from {}\n{}'.format(
Frigate | 2024-05-09 18:02:29.646251030 ValueError: Failed to load delegate from libedgetpu.so.1.0
Frigate | 2024-05-09 18:02:29.646252291Operating systemOther Linux Install methodDocker Compose Coral versionUSB Any other information that may be helpfulNo response |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 12 replies
-
|
one user saw this issue and it was fixed by deleting the model in model_cache and then restarting frigate |
Beta Was this translation helpful? Give feedback.
-
|
Good idea but it did not work :( |
Beta Was this translation helpful? Give feedback.
-
|
this shows that the coral was found but then is getting stuck. Are you using the correct edgetpu model and not the cpu type? |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, though I just tried the CPU one and it runs, though the Coral TPU is hovering around 100%: I figured maybe the models were uploaded manually and incorrectly tagged but 100% for a single camera does not seem right seeing that it was under 10% with the non-plus model so that is probably not the case. EDIT: And the inference time is way above normal using the cpu model. |
Beta Was this translation helpful? Give feedback.
-
|
something seems off, not sure why it would be having problems if it is loading the model correctly. @blakeblackshear may see something I am missing |
Beta Was this translation helpful? Give feedback.
-
|
Maybe I could try with the 'vanilla' 2024.1 model if there is a model ID, in case something got introduced in the custom model? |
Beta Was this translation helpful? Give feedback.
-
|
Is this behavior consistent? Is the timing of failure always about the same? |
Beta Was this translation helpful? Give feedback.
-
|
It happens every time I start up the container, and it seems to happen right away when the inference starts. The only thing that isn't consistent is that sometimes it will simply hang, then be killed, then it retries, hangs and gets killed on a loop while other times the Coral seems to completely crash and I get the driver error. I can run the non-plus model for days though without any issues. |
Beta Was this translation helpful? Give feedback.
-
|
What is the model id? |
Beta Was this translation helpful? Give feedback.
-
|
653f750e7e295ca1496859731ffa77ef FYI this is the edgetpu I am using : https://www.mouser.ca/ProductDetail/212-842776110077 Thanks for helping! |
Beta Was this translation helpful? Give feedback.
-
|
So I can confirm that this model works just fine on my machine with the same USB Coral. It must be something specific to your hardware. Does this happen with both of your models? |
Beta Was this translation helpful? Give feedback.
-
|
The CPU model works but the inference speed is 550ms (I posted some stats in my thread with Nick above). Since you ruled out the model I tried a couple of things on the PC and ended up noticing it does not crash on USB2.0. Thank you ;) |
Beta Was this translation helpful? Give feedback.
-
|
This sounds similar to issues others have had due to lower power on the USB port or a bad USB cable. The Frigate+ model is faster than the default model, so data will be going in and out at a higher rate. |
Beta Was this translation helpful? Give feedback.
-
|
I will try swapping out the USB cables as well. Thank you for the quick help and great project! |
Beta Was this translation helpful? Give feedback.
So I can confirm that this model works just fine on my machine with the same USB Coral. It must be something specific to your hardware. Does this happen with both of your models?