How to fix false positives
I have a tree in the backyard that creates a shadow in the late afternoon that is throwing a ton of false positives. Like 2 to 3 per minute. Enough to generate an error of "Motion-triggered emails are being sent too frequently and so will be rate-limited." Given that it is the same camera that is having the same issue on sunny days (we get a lot of them), it would be nice to train the AI not to get tricked.
Comments
I wonder if the camera has WDR options in its software configuration. I’ve had luck using WDR to remove shadows in my new Hikvision cams. I set it to auto in WDR but I had to increase shutter speed to allow this to around 1/30. Its default was set at 1/6.
In the last 24 hours, I'm having the same issue on more than one camera. No amount of changing settings seems to make a difference. In my case it seems to be because of changing cloud cover. It is constant. I have human and animals selected in settings. That means I should JUST get notifications when it detects one of those and not motion in general, correct? The notifications often just say "motion", not human or animal detected. I have had to just disable things until I can spend time troubleshooting. It's been fine for months, and I don't remember having to deal with this last spring. I remember occasionally having to adjust a mask because of vegetation and strong summer shadows or wind. I could be wrong, but It really feels like something is fundamentally different.
@macluddite - if you are getting notification that say "motion", this means that the AI object detector is not enabled for Actions mode, which covers notifications. Go to Settings > Cameras > Triggers and make sure that one of the AI options (Humans / Vehicles / Animals) is enabled for Actions mode. With one of these options enabled, you should never get a notification that says just "motion", because motion alone is not sufficient - one of the selected objects will have to be detected in addition.
@schneideris - this could be the problem you're seeing too, so please verify the above.
I would also recommend going to Settings > Triggers > Advanced Options, and verifying the following:
Part of the problem here is that the current motion detector is susceptible to extraneous movement like shadows, snow, rain etc., so even with the AI object detector enabled, some false-positives still get through.
That's why we have been working hard on the next generation of motion detection for SecuritySpy, which uses a bespoke neural network to much more accurately determine if there is valid motion happening or not. This is still in development, but we would be grateful for anyone to test this and report back their results. This new features is included in the latest beta version of SecuritySpy (currently 3.10b5).
Note that this enhancement is available only on Mac with Apple Silicon chips, not older Macs with Intel chips. This is because it relies on the Neural Engine built into Apple's chips. When used on a Mac with an Apple Silicon chip, the new motion detection neural network will be enabled automatically. Please report back your experiences if you are able to test this enhancement.
Here are my settings for the camera. Perhaps I should change the motion detection sensitivity. If so, should I increase or decrease?
Hi @schneideris your settings look fine, I wouldn't recommend changing anything. I suggest you try the new beta that I linked to - it is now highly resistant to tree shadow movement. If you have an Intel Mac, there's an option to enable the new motion detection in the advanced trigger settings for specific cameras. However, on an Intel Mac, this should only be used on relatively recent/fast machines, as it may use significant CPU resources (on Apple Silicon, all processing for this is done in the Neural Engine, so it should actually use less CPU resources).
Thanks @Ben, installing now and will let you know how it goes. Today might not be a good test, but the forecast for tomorrow afternoon looks promising to see what happens.
@Ben Just thought i'd feedback, i updated to SecuritySpy beta version is 6.10b6 48 hours ago on my M1 MacMini and initial testing is seems really good, shadows and spider webs at soffit level have regularly caused false detections via stationary vehicles at night, however when i reviewed this morning they've been mostly eliminated, around 20gb less wasted storage so far, very impressed!
On a secondary note, i have no use for animal detection with my setup, however If i could only reduce animals (cats, dogs and birds) from getting flagged as humans, i haven't tried raising the Human AI detection threshold yet, as i wouldn't want to risk missing genuine triggers. Goes without saying I love what your doing, whenever i think the software can't be improved upon it is, v6 has been a game changer.
Apologies @Ben as I had not yet responded. Uncharacteristically we have not had the really bright days to make it definitive, but at this time I would say that the beta has cured the false alarms from shadows.
@JC_5674, maybe you want to turn on the animal detection and then just don't do anything with them. Total guess but with animal detection turned off it might be a thing of the first line of defence has been turned off so the next line thinks movement of that type is somehow human. Just a guess though.
Great to hear the new motion detection is working well - thanks both of you for reporting back. We're just putting the final refinements to this and hope to release it soon.
We're also working on retraining the Human/Vehicle/Animal classifier with an enhanced dataset, which should make it significantly more accurate. This may be in the upcoming 6.10 release, but if it's not ready in time, should be in the subsequent 6.11 release. This should make animal detection in particular significantly more accurate, making animals less likely to be flagged as humans.
@JC_5674 one thing to be aware of is that if you have stationary vehicles in the camera's view, you need to make sure that these are entirely covered by the motion mask, or if this is not practical, you can't use the vehicle AI classifier option. The new motion detector will improve this situation (because many fewer false triggers like spider webs will make it through the motion detector to the classifier), but it won't eliminate it completely.
@Ben, it is back. I was still on the beta but just had a raft of false positive emails. I upgraded now to the full 6.10 release and will let you know if anything pops up with it.
Hi @schneideris do you have an Intel Mac? If so, you may need to go back to Settings > Cameras > Triggers > Advanced Options, and make sure this camera is set to the new AI motion detection method.
The new AI motion detection has now been officially released in SecuritySpy version 6.10. For anyone who wants to read more about how this works, please see our new blog post entitled AI-Powered Motion Detection.
@Ben, I have a Mac Studio with the Apple Silicone M2 Max. Read through your article and did the AI debug test with stellar results. It is on Automatic. Do have a question on thresholds that I posted as a comment on the article. Basically wanting a better understanding of what the slider means. If at 100 does that mean the reporting of an Animal would only happen if the system is 100% sure it is an Animal? If set at 25 then more potential for something to be considered an Animal when it is not?
Your understanding of the AI sliders is correct: if you set the Animal slider to 100, then you will only get a trigger when the system is 100% sure that there is an animal. Such a high degree of certainty won't happen very often, so this would result in far fewer animal recordings/notifications (including quite a few missed animals), however the ones that you do get will probably all be correct detections.
It's best to leave these sliders on 50, unless you are having a particular problem with some object being erroneously detected too frequently when it's not actually present (in which case you could increase the threshold to 60-70), or if you have a problem with an object not being detected enough when it actually is present (in which case you could decrease the threshold to 30-40).
👍️
Just upgraded and am excited to check this out. I have had to turn off actions on all the cameras during rain or snow because the emails were non-stop. And I get the sun casting shadow emails as well although not nearly as much of an issue.
Definitely looking forward to seeing how the animal detection does. I know the animals are looking forward to it as well....
EDIT: I can already tell an improvement. On one camera, whenever I would pan it around it would always generate a motion alert. But I just tested it out and it did not generate any alerts when panning. I can't believe I'm saying this, but now I'm looking forward to the rain, snow, and next wind storm to see all those false positives go away. This is a huge improvement, many thanks for the work!
Motion has been good at detecting bears, which is my primary use case (I've had one on the deck everyday for the past two weeks and use your software alerts to shoot them with rubber slugs). Your software would detect them before they got onto the deck which is great. But motion came with the price of the false alerts.
I have noticed with the new AI that I'm not getting the animal alerts that I would expect. I put the camera on some birds and it took longer than I would have thought given the motion to get an alert. The slider was set to 50.
I'm also having the same issue with squirrels, but I set the slider to 30.
Also no alert on a snowshoe hare at slider 30. I followed him around and had better images with plenty of motion, but no alert.
I don't care about birds or squirrels or hares (ideally, it would be great if your software returned the label and confidence level for each detection and users could check boxes of animals they wanted alerts for) but I do care about bears and moose. I'm going to use a bit more caution on the detection while we see what it does. I've opened the door and kicked up nearby bears before and would like to avoid that...
I might try taking that slider down below 30 as well.
Setting the slider to 10 and the motion to single frame seems to be working much better. Picked up the squirrel quickly and picked me up sooner than normal walking to the deck.
It also picked up a hare quickly.
Even picked up the hare from a distance.
I feel confident this will pick up bears and can't wait to see a lack of false positives!
Didn't take long to confirm it finds bears.
I thought it had a few sunlight/shadow alerts but turns out it didn't. One was a bird and the other was a squirrel. Looking forward to seeing how it does in the rain and snow and wind. This is very promising so far.
@AlaskanOffGridder, I liked my use case of detecting package deliveries, but am now thinking I am a rank amateur in comparison to your use case and related automations. Nicely done!
We definitely don't get any deliveries out here, maybe one day we'll get drone deliveries. But these bears are non-stop. I've got to get my electric fence working again. And I'm trying to crack the HTTP Request code on these Dahua cameras so I can have my python code control them and do some other things like play outdoor speakers and fire off an air cannon, but so far the only HTTP Requests in the API documentation that work are pan up, pan down and stop. Nothing else works.
Maybe I need a package delivery of how to program better speaking of being a rank amateur...
So far no false alert during a nice little shower with some sun streaming.
Have you looked at HomeHelper to trigger things based on SecuritySpy actions/events? It is HomeKit centric but works quite nicely. With all the AI being applied everywhere I can see HomeKit growing in capability (if Apple can get some decent guidance and development done).
I haven't. I'm hoping to stay with my python code since it runs so much other stuff on my homestead.
Fair. Simpler is better. Not relying on Apple or others to maintain bug free and stable platforms is priceless.
Great to have your feedback @AlaskanOffGridder. I would recommend minimum 0.5s as the trigger duration for this outdoor camera - 1 frame will probably give you too many false-positives, even with the new AI. Small animals like squirrels are going to be a hard case, as will be the hare, since it is really quite well camouflaged against the background. In addition, we don't have too many squirrels and hares in our training dataset, so the detection accuracy for these isn't going to be as good as more common and/or larger animals. However we are actively trying to increase the number and variety of animals in our dataset to increase the accuracy of their detection, so you can expect this to improve over the next few versions of the software.
Thanks Ben, much appreciated. I will tweak the settings but have to say so far it has been working great. I don't think I've had any false positives and it's amazing how well it picks up the squirrels despite the limited dataset training.
This is a huge improvement.
Finally got some false positives.. some sun streaming false positives, changed the trigger duration to .5 as you suggested and they went away.
It was windy and it poured all last night and still is. In the past I would have shut the motion off because it would have been non-stop emails and alerts. With the new version, there wasn't a single false positive.
Great that you're having such good results @AlaskanOffGridder - thanks for your feedback here. Ignoring brightly-lit rain streaking across the frame was one of our big priorities when crafting the new algorithm, so I'm very glad to hear this.