[Idea] Zero-shot object detection models?

[afaulconbridge]

Describe what you are trying to accomplish and why in non technical terms
I want to be able to use zero-shot object detection / zero-shot image classification so that I can record wildlife. IR footage is poorly supported and many training data sets are poorly labelled for uncommon animals (fox or hedgehog for example).

Describe the solution you'd like
I'm curious if anyone has considered this before, and if so what pitfalls there might be. Or if there are any plans for "plug ins" or similar ways to extend Frigate. I'll have a go at my own fork with an object-detection implementation, and I might be able to get something working locally for my use case, but I probably won't be able to generalise it without help.

Describe alternatives you've considered
I could train my own model (Frigate+ style) but I'd rather not as its a lot of work to prepare, update, etc. Zero-shot approaches seem to be improving faster, and off-the-shelf models improving too.

Additional context
I've been building my own system from scratch, but then I discovered Frigate and it already does so much of the "boilerplate" much better than I could. Nice job!

[NickM-27]

Frigate already works with deepstack (CodeProject.AI) so that would work for many cases, Frigate also supports using any model you like as long as it is compatible with the detector of your choosing. I am not sure what exactly the request is here

[afaulconbridge]

Deepstack, and the other models in Frigate as far as I can see, work with pre-defined and pre-trained labels. So it can never detect a hedgehog, because it has no concept of what a hedgehog could be, because there was no hedgehog in the training labels.

Zero-shot models are only given the categories at execution time, and can work with things that they've never seen before - similar to large-language models. Broadly, a zero-shot model knows that a hedgehog is a small spiky spherical mammal, so when it sees a picture of a hedgehog it can give the right answer - even if the training data didn't contain any hedgehog pictures and only included picture of things that were either small or spiky or spherical or mammals.

I'm going to mess around with using zero-shot models in Frigate. There's no IRC, discord, reddit, email, or other contact/community details I could see - so a GitHub issue it is. I know Frigate is MIT licensed, so technically I don't need to tell you, but I thought it'd be nice to say something (and might save me a bunch of time if it was trivially impossible for some reason). Plus if there was anyone else who has/is/will try this it might be useful to cooperate.

[fgoodwin]

https://www.reddit.com/r/frigate_nvr/

Additional tags are coming Eventually™ but I also am interested in more generalized detection for wildlife, so if you get it working definitely write something up.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.