Frigate+ Base Model 2024.1 Update

[blakeblackshear]

NOTE: The base model was updated 24 hours ago. If you recently requested a new model, check the base model version before requesting a new one.

The base model in Frigate+ has now been updated to 2024.1. All model requests going forward will use this version. In order to get a new model, you will need to submit a model request.

In addition to incorporating new images from Frigate+ users, this base model update was focused on reducing known false positives. To date, nearly 250,000 false positive examples have been submitted by users. The previous base model reproduces 26% of these false positives and the new base model reduces that to 11%.

I also updated the fine tuning process to increase the effect that your verified images have on your model. My fine tuned model now reproduces only 1.5% of my reported false positives.

Here are some stats showing the false positive reduction for my model.

Model	Results
2024.0 Base Model	Reproduces 18.6% of my reported false positives
My Tuned 2024.0 Model	Reproduces 16.7% of my reported false positives
2024.1 Base Model	Reproduces 9% of my reported false positives
My Tuned 2024.1 Model	Reproduces 1.5% of my reported false positives

According to every objective measure I have looked at, this should be a significant improvement in false positive rates, but I expect results to vary by user. I would appreciate some user feedback (both good and bad) after upgrading.

[Whomever6378]

Very excited to try the new model, there has been one plant in particular out the front of my home that supposedly looks exceptionally human 😅
I had actually just submitted a new model request maybe 5 hours ago?
Will this happen to be on the new 2024.1 or should I expect to submit another request for the new base model?

[blakeblackshear]

Any model requests submitted in the past 24 hours will be using the new model.

[YBonline]

Although ideally the AI should figure it out, if its one plant in particular, you can mask that object on that area. I have multiple tables with umbrellas, and I've had to mask those spots because Frigate is very insistent the umbrellas are people with extremely high confidence even after training tons of photos..

[Benoulimou]

Hello ,
Thank you for all your work on Frigate. I use it all the time as a complement to other things while I refine it and I really hope to succeed in using it exclusively!
However, is it possible to add an option for multiple deletion of images sent to Frigate+ in order to redo models more in line with recent settings (change of cam position for example). It would make things a lot easier.
Oh yes, and is it possible to delete models?

Sorry for the questions, thank you very much for all your work! I can't wait to try it out!

[NickM-27]

If you move a camera you shouldn't be deleting old images, I've done this multiple times without any issues

[Benoulimou]

There are really a lot of them in the old position, won't that "disrupt" the model?

[NickM-27]

No it shouldn't. I've moved my garage camera twice and even took down a camera that had a few hundred images and it hasn't caused any issues at all. Blake has also answered that it shouldn't cause any issues in most cases.

[Benoulimou]

thank you for your reply

[RWMoo]

Very excited to try this, really appreciate the hard work you put into this. I'll provide feedback over the next week :)

[bbrodriguez]

I’ve been eagerly awaiting this, 2024.0 was great and set a high bar.

Just got my 2024.1 completed and configured in and super keen to see how it fares.

edit: I’d love to see a write up / summary of what changes you made to training sample selection this time.

[blakeblackshear]

Very similar to the last round, but I weighted the sampling to focus primarily on false positive examples.

[bbrodriguez]

My initial impression is that it is an improvement. I dropped my thresholds a little based on the stated reduction in false +ve’s - to try to increase detections through the night.

Of the images below, I might have expected at least one cat detection to have been labelled a dog under 2024.0.

Admittedly it was a clear night so not challenging images to process.

It still seems strong for people at night also which is important to me.

[bbrodriguez]

So a few false +ve detections creeping in, but I can bump my thresholds back up a little (dialled them back to 0.75) which would eliminate many.

What I do think would help is improved motion detection (which I understand is in the works) - specifically tracking an object through its motion. I seem to suffer occasional bouts of a table being detected as a dog while a dog is in the frame somewhere - maybe after it passes by.

[NickM-27]

specifically tracking an object through its motion

frigate already does this, a false positive dog and the real dog are most likely unrelated unless you see them as the same event when using the recordings timeline viewer

[blacknell]

I'm wondering how we could get false positive statistics for our own data - like you summarised in your intro.

[blacknell]

Excellent

[blacknell]

If it's relatively straightforward would be good to get a view of how many false/true positives/negatives there are by label/camera so I can focus on adding those to frigate+ ready for next model request. with 400+ images it's getting difficult/becoming guesswork as to where to focus

[NickM-27]

I don't know that the goal should be even label distribution. I think the goal should be focusing on where defections are missing or boxes are inaccurate

[blacknell]

Fair point but it would show gaps by label

[pyrodex]

I wrote a simple Python script that does this for my needs, it allows me to submit true/false by label, camera, zone, or a combination of all three. It also allows me to purge events based off those factors if I ever decide to do so.

~$ frigate-events.py -h
usage: frigate-events.py [-h] [--silent] [-y] [-p] [-s] [-n] [-f] [-t] [-l LABEL] [-c CAMERA] [-z ZONE]

Utility script to help process Frigate events in large amounts

options:
  -h, --help            show this help message and exit
  --silent              Disable verbose output
  -y, --yes             Enable automatic YES instead of being prompted
  -p, --purge           Purge events
  -s, --submitted       Process submitted events only
  -n, --not-submitted   Process not submitted events only
  -f, --false-positive  Submit the events as False Positive to Frigate+
  -t, --true-positive   Submit the events as False Positive to Frigate+
  -l LABEL, --label LABEL
                        Specify Frigate Label, these are always lower case (e.g. person)
  -c CAMERA, --camera CAMERA
                        Specify Frigate Camera, these are the same CASE as displayed in your Frigate configuration
  -z ZONE, --zone ZONE  Specify Frigate Zone, these are the same CASE as displayed in your Frigate configuration

[mintjberry]

Honestly I was very underwhelmed by my first model using 2024.0, even after tuning it wasn't a lot better.
Am excited and hopeful for 2024.1, have requested my first model using it, thanks.

[rough316]

Tried my first + model, its not working. I get this in the logs:

2024-04-26 03:19:54.587111853 Process detector:ov:
2024-04-26 03:19:54.587125103 Traceback (most recent call last):
2024-04-26 03:19:54.587134163 File "/usr/lib/python3.9/multiprocessing/process.py", line 315, in _bootstrap
2024-04-26 03:19:54.587142563 self.run()
2024-04-26 03:19:54.587150983 File "/usr/lib/python3.9/multiprocessing/process.py", line 108, in run
2024-04-26 03:19:54.587159223 self._target(*self._args, **self._kwargs)
2024-04-26 03:19:54.587171553 File "/opt/frigate/frigate/object_detection.py", line 125, in run_detector
2024-04-26 03:19:54.587179833 detections = object_detector.detect_raw(input_frame)
2024-04-26 03:19:54.587196023 File "/opt/frigate/frigate/object_detection.py", line 75, in detect_raw
2024-04-26 03:19:54.587204733 return self.detect_api.detect_raw(tensor_input=tensor_input)
2024-04-26 03:19:54.587212953 File "/opt/frigate/frigate/detectors/plugins/openvino.py", line 84, in detect_raw
2024-04-26 03:19:54.587233863 infer_request.infer([tensor_input])
2024-04-26 03:19:54.587242323 File "/usr/local/lib/python3.9/dist-packages/openvino/runtime/ie_api.py", line 152, in infer
2024-04-26 03:19:54.587251023 return super().infer(normalize_inputs(self, dict(enumerate(inputs))))
2024-04-26 03:19:54.587259413 RuntimeError: The input blob size is not equal to the network input size: got 307200 expecting 270000

[NickM-27]

Process detector:ov:

You're trying to use openvino which isn't currently supported

[rough316]

Was thinking thats the case, trying to get it working with CPU, then will go find my USB Coral.

[rough316]

Up and running on the Coral USB. You guys are doing a fantastic job on Frigate!

[Sch0field9]

I have statues of lions on my gate pillars, which are often mistaken for either a person or a dog. (Interestingly, they are almost never mistaken for a cat.) From the camera's perspective, people almost always walk behind the statues.

You mentioned that to reduce false positives, we should submit true positives in that area along with the false positives. Now, if a person walks behind one of these statues and I submit it as a true positive with the statue overlapping the person, would that confuse the model?

[Sch0field9]

Regarding that topic: the camera in question is a PTZ camera, making it impossible to mask out the statues. Is there a solution to this issue, or is something planned for zones/masks on PTZ cameras?

[blakeblackshear]

As long as the person is actually visible too, it should be fine.

[blacknell]

So in this circumstance where the Frigate+ identified box is too big should we mark this not a package and draw a tighter bounding box?

[YBonline]

I believe you say it is a package, and then adjust the size of the box before verifying

[bbrodriguez]

I’ve noticed the face boxes are extremely large in 2024.1.

[NickM-27]

Yes, this would be this is a package and fix the box.

Face boxes look accurate just like always in my usage so far

[YBonline]

So my initial results are that false positives are indeed way down, but false negatives are up as well (which is obviously a bit of a bigger deal). This might be because I hiked up my min score on the previous model, so I'm gonna try to reduce that and see how it goes.

The camera with the most notable negative impact in my sole newer (to me) fisheye camera.... its detecting near nothing now. I don't have a whole lot of training on it either, so I'll have to see if I can drop the minimums pretty low and start training it. I do have it using the raw stream, with no de-warping, so I don't know if that is a proper goal or if there should be a different model for this unusual camera format?

I just didn't want to create multiple additional streams as I have so many cameras already, and I'm near the limit of what my hardware can support without more RAM.

[nshp]

Looking pretty good so far with the new model. Still some false positives and misidentifications here and there, but the false positives have been noticeably reduced, making it much easier to scroll back through events. The only identification that doesn't seem to have been reduced is deer, but most of those are relatively low confidence, I'll just set the minimum higher. The stationary object tracking seems to be working much better for me with parked cars. It was losing the car-ness of parked cars many times a day previously, now it's pretty solid. I frequently have two cars that overlap from the camera's perspective, I could watch the detection going nuts hopping back and forth between which one of them was a car and which wasn't.

I did notice one particular type of new false positive showing up. One of my cameras can see the street through some (sparse) tree branches. I've been submitting plenty of true positives of cars driving by seen through the branches, Frigate has been doing a fine job of catching those. Now it seems I've trained this new model that cars, and branches blowing in the wind, are cars. I'm submitting those false positives now, hopefully it'll go back to normal with the next model I train.

I think my biggest remaining problem model-wise is that squirrels and foxes are always running by firing off a confident stream of "cat - dog - cat - dog - cat - dog" events.

[NickM-27]

Some of these sound like things that the new UI will greatly improve the experience of.

Stationary tracking has also been improved for the next version.

[krustophenia]

Thanks for your excellent work!! One question, I've submitted some images with dog labels from different cameras, day/night etc but my frigate is not detecting any since I started using this custom models. Should I keep uploading and labeling dog images? I've lowered the score for dogs to 40-50% and even like this no detections. Thanks.

[NickM-27]

You should include your config, seems like a config issue

[krustophenia]

model:
  path: plus://XXXXXXX
mqtt:
  host: XXXXXXX
  user: XXXXX
  password: XXXXX
detectors:
  coral:
    type: edgetpu
    device: usb
cameras:
  Cam_Piscina:
    ffmpeg:
      inputs:
        - path: rtsp://XXXXX:XXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
        - path: rtsp://XXXXX:XXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - rtmp
    motion:
      improve_contrast: true
  Cam_Passadis:
    ffmpeg:
      inputs:
        - path: rtsp://XXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
        - path: rtsp://XXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - rtmp
    motion:
      improve_contrast: true
      mask: 
        - 389,203,402,0,1280,0,1280,576
  Cam_Parking:
    ffmpeg:
      inputs:
        - path: rtsp://XXXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
        - path: rtsp://XXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - rtmp
    motion:
      improve_contrast: true
  Cam_Frontal:
    ffmpeg:
      inputs:
        - path: rtsp://XXXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - detect
        - path: rtsp://XXXXXX@XXXXXX:554/cam/realmonitor?channel=1&subtype=0
          roles:
            - record
            - rtmp
    motion:
      improve_contrast: true
  Cam_Menjador:
    detect:
      width: 1920
      height: 1088
    ffmpeg:
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: rtsp://XXXXXX/live0
          roles:
            - detect
            - record
    motion:
      improve_contrast: true
  Cam_Carrer:
    detect:
      width: 640
      height: 480
    rtmp:
      enabled: False 
    ffmpeg:
      input_args:
        - -avoid_negative_ts
        - make_zero
        - -fflags
        - nobuffer+genpts+discardcorrupt
        - -flags
        - low_delay
        - -strict
        - experimental
        - -analyzeduration
        - 1000M
        - -probesize
        - 1000M
      output_args:
        record: -f segment -segment_time 10 -segment_format mp4 -reset_timestamps 1 -strftime 1 -c:v copy -c:a aac
      inputs:
        - path: rtsp://XXXXXXX@XXXXXX:554/h264Preview_01_sub
          roles:
            - detect
            - record
    motion:
      improve_contrast: true
    objects:
      filters:
        dog:
          min_score: .6
          threshold: .7
        person:
          min_score: .70
          threshold: .75

detect:
  width: 1280
  height: 960
  fps: 3
  enabled: True
  max_disappeared: 25

record:
  enabled: True
  events:
    pre_capture: 5
    post_capture: 10
    retain:
      default: 10

snapshots:
  enabled: true
  timestamp: true
  bounding_box: false
  retain:
    default: 10

timestamp_style:
  position: "br"
  format: "%d/%m/%Y %H:%M:%S"
  color:
    red: 0
    green: 0
    blue: 0
  thickness: 1
  effect: "solid"

birdseye:
  enabled: True
  width: 1280
  height: 720
  quality: 1
  mode: motion

objects:
  filters:
    dog:
      min_score: .4
      threshold: .5
    person:
      min_score: .80
      threshold: .82  

[NickM-27]

You haven't added dog as an object for frigate to track (and you also haven't added face, losing a feature of frigate+ model as well)

[krustophenia]

oh right, my bad. Adding this and let's see if that improves. Thanks!

[csreynolds]

At risk of resurrecting a horse to beat; was a list of labels for future models firmed up?

I'm keeping my raised vegetable gardens squirrel free with homeassistant, a smart plug and powered water valve to spray them if frigate detects them in frame.

Currently the squirrels are detected as birds, so I am hesitant to update the model without a squirrel tag or something to keep this working as it is currently.

Not a complaint in any way, this project is great and all your hard work is very appreciated. 🥇

[nwish]

No changes to the detectable objects that I can see.

[blacknell]

I'm seeing (subjectively) a huge drop in false postives particularly at night. I'm also seeing a small number of new false positives (new in new places). So I guess this is 10 steps forward and 1 back. Brilliant.

[OnFreund]

Same here

[dz4news]

Same here... I have actually dropped the min_score on a few objects and still have very few false positives.

[OnFreund]

I'm thinking of lowering min_score too, to avoid cases where one logical event is split into many shorter ones. Not sure if that's needed as much with 0.14 though.

[krustophenia]

Regarding face detection, I've added face as object:track and added

face:
min_score: .80
threshold: .82

shouldn't that do the trick?

[NickM-27]

You won't get face events, just that face will be labeled and used to choose the best image for person

[nwish]

~48 hours on 2024.1 and this is the only false positive I’ve seen yet… a friendly spider-person 😂 but a bit before that, we did get a nice unexpected true positive (100% confidence?? 🧐) of the Jeep down there, despite the obscurities.

So far so good!!

I have 6 cameras with roughly 2500 images submitted.

[nwish]

Two weeks on 2024.1 and still the only false positive has been Mr. Spiderman above. This has been with min_score and thresholds set at 75 and 85 respectively for most objects. I just lowered them all by 5. We shall see!

[vuminhkh]

Hello, I just like to add a feedback for this new model, it's way better than the one last month in false positive and it detects really better also at night. I have 2 cameras at the front of the house, usually only one camera can catch me at a time, now basically the two cameras will catch me in the same time wherever my position is. I still have some false positives, but it's really way better than before ! I'm just having the feeling that if more frigate users upload and train the model, it'll improve more and more everyday. Well done !

[dimatx]

My subjective experience with the new 2024.1 is very good and the improvement over 2024.0 is substantial. I was able to lower my thresholds, and still getting noticeably lower false positive rate.

[gacekk]

Hi,

I just got my model based on newest version. 10 cameras were trained with average of 15 images each and I am getting a lot of false positives. As an example, for below (camera was trained in 30 images) the model wrongly detected person, and did not detect a car. I will be now gathering and sending all those false positives and generating new model in next days.

[blacknell]

I believe the guidance is minimum 100 images per camera. Ideally mixture of legit and false positives. And variety of lighting circumstances

[dimatx]

Also, you may benefit from setting max size for the person object as well as masking the trees so that people are detected only if they are on the driveway surface.

[kirsch33]

agreed with all others here, substantial improvement with 2024.1 over 2024.0. this one is almost perfect

[adamburgoyne]

Are models applied to all cameras or individual ones? My doorbell has worked perfectly for the past year but a new camera on the side of my house is throwing up false positives nearly daily - 76% chance of a plant pot or garden chair being a person sort of thing where I expect it to be mostly silent on the notifications. I have been uploading a load of positive/false positive verified images in hope that a model might solve it and I can use it more like an alarm notification. I don't want to cause any problems for the perfectly fine doorbell camera though!

[NickM-27]

it is applied to all cameras, I wouldn't really see any issue as doorbell cameras are quite common so there are likely many examples in the base model with that camera perspective

[gacekk]

I have similar experience. Out of my 10 cameras, half are showling lots of false positives, while the other half are having none or close to none. While all are in fact in same environment (e.g same types of plants). I checked and this is not related to how many images were used for model training on my end.

[adamburgoyne]

That's good to hear, I was hoping to avoid having to train my own for the doorbell but I guess it wouldn't hurt if I really had to. I was holding off until this update but everyones reporting good things with very high confidence scores.

At the moment I never really see much higher than 85% (normally 70-80%) even if I'm stood right in front of it and sometimes the chairs are 75% a person (so even setting confidence level higher at the moment is pointless) so hopefully this will help.

[NickM-27]

the default coral model has a maximum possible score of 84%, to be clear

[adamburgoyne]

That would explain that then, just checked and yeah 84% on the latest person event not quite 85%! Then right below it the plant pot at 84%. Fingers crossed they both don't shoot up into the 90s!

[kirsch33]

one question that I dont see addressed in the docs--do we need to label objects in submitted images that we dont intend to track in our own setups?

For example, its very cumbersome to label licenses plates in many of my cameras but this isn't an object I'm interested in tracking and I dont have it configured. Should I be labeling it in Frigate+?

[adamburgoyne]

I only have the option to label objects I've set to track. I can only see cat, dog, person in the UI dropdown when labelling my submitted images.

[NickM-27]

that is configurable in the frigate+ settings page for each camera. By default it is all objects

[kirsch33]

thats part of why i was asking--if by leaving it out I'm creating more work for Blake to have to add back in the license plate label if he decides he wants that image in the base model then I dont mind at all. but if it doesnt matter either way then Id prefer not to.

[NickM-27]

@blakeblackshear would be best to provide the final word on that

[blakeblackshear]

You don't have to label things you don't care about on any of your cameras. It would probably create a problem if you wanted to track cars on one camera and left a bunch unlabeled on another camera though.

[theyapps]

I have a pretty human looking lamp.

I can see why this has been so difficult to convince the model is not a person. I sent about 300 images so far and only had mild success. This new base model has made worlds of difference. So far after about a week I have had zero false positives!

[gacekk]

Question on Send to Frigate+. If we use this option, and then go over the Positive matches, should we correct the bounding boxes to be more fitting with actual objects? making them smaller to fit better. I guess that in case where its missing and not catching entire object we should absolutely correct it.

[gacekk]

One more thing that just came to my mind. Let's assume I have tuned the model for all my cameras' substreams that are either 640x480 or 1280x720. Then I decide that the detection is missing some elements and switch the detection to main streams which are 4k-8k resolutions?

[NickM-27]

it would not be an issue because the regions are resized to 320x320 anyway. Though I personally would not recommend running detect at larger than 1920x1080

[gacekk]

That would be due to hardware impact?

[NickM-27]

yes

[gacekk]

OK, thank you. I am now preparing the recommended 100+ images for all my 10 cameras, will request model after that and see what will be the results

[phillym]

I haven't had a single false positive with 2024.1. With 2024.0 I got heaps. I ended up churning through model credits trying to train out the FPs, with mixed success.

Will you make additional model credits available once we hit 0 available? I went through 7 or 8 semi-futile model credits with 2024.0.

[blakeblackshear]

Yes. Just reach to me via email if you run out of credits.

[adamburgoyne]

Hey @blakeblackshear, just hijacking this comment.... What's the turnaround of getting access after sending over an email? Not in a rush but not sure if it's an automated process or manual your end!

[blakeblackshear]

I am working on a few changes that I want to finish before sending out more invites. Normally it would be within a day, but I'm expecting to send out any outstanding invites this weekend once I am done with the changes.

[adamburgoyne]

No problem, I'll hold tight then! Thanks

[mintjberry]

2024.1 is working for me far better than 2024.0 ever did. Much happier with it. Still getting the odd false positive (will retrain on a new model soon) but noway near what it was.

[adamburgoyne]

I've just trained my first model (about 600 verified pictures) and so far so good, it's detecting all the things it was previously with higher scores and hopefully will continue with less false positives - nighttime is normally my cameras weakness. I did have a false positive earlier (rare during daytime on the standard model) and I can't figure out why it would remotely think it was a person (72% in the UI, 67% in the snapshot) but after reading some comments above it sounds like I might at least benefit from setting a min_score and threshold which I've not set yet. Is there a good starting point for these as I don't really know what to start off with, I've seen this listed above:

      min_score: .80
      threshold: .82  

[adamburgoyne]

I've just found you can see ratio in the clip playback, I assume that's related but I've set to 0.7 and 0.75 for now. Will see what it picks up on.

[NickM-27]

Ratio is the ratio of width to height of the box. There is a section of docs that answers your questions exactly https://docs.frigate.video/plus/first_model#step-4-adjust-your-object-filters-for-higher-scores

[adamburgoyne]

Ah ha! I'd read the following page on "Object filters" but not that specific Frigate+ one. I'll start mine a little lower and go up instead of the other way around as I'd prefer to be alerted more than less!

I see in some of those there's no threshold specified at all? Is there a default one or does it take the min_score value?

[NickM-27]

Threshold doesn't apply to the attributes / sub labels

[adamburgoyne]

Got it, thanks!

[krustophenia]

I saw an improvement with the new model but I got some false positives that I already labeled on frigate+ a few months ago. Is that normal? Will I need to label things again when base model changes?

[miziu]

Hello, is the release date of the version 2024.2 known?

[blakeblackshear]

Updates will be roughly every 3 months.