False positives in event stream

benish

I've got SecuritySpy connected to Home Assistant, via SecuritySpy for Home Assistant and have it configured to send me a notification when a human is detected. I seem to get notifications of actual humans as expected, but am also getting a fair number of false positives (it says a human is there, but there isn't). Strangely, SecuritySpy doesn't capture a video clip for these events, but it does send an event to Home Assistant.

I have Motion Capture and Actions AI triggers set to the same value: https://go.benish.io/lvICO9 although I believe SecuritySpy sends events based on Motion Capture, regardless of the Actions settings (i.e. if I turn off Actions in SecuritySpy I still get the events in Home Assistant).

1. Any ideas why SecuritySpy would send a "human detected" event, but not capture a video clip? As far as I understand, it should do both based on the settings I have. This is not a big deal if I can reduce / eliminate the false positives, but seems like something is wrong here.
2. Is there any way to see what score a human detection event is getting in SecuritySpy? I assume to reduce these false positives I need to raise the Human threshold setting, and it'd be great to know what score all the previous false positives have had so I don't have to guess at the appropriate value.

Ben

SecuritySpy sends the following related events to the event stream:

MOTION: this signifies motion of any kind, and includes four coordinates that describe the location of the motion within the image

CLASSIFY: a frame has been classified by the AI, and the percentage probability of both a human and vehicle is included, e.g. "CLASSIFY HUMAN 80 VEHICLE 0"

TRIGGER_M: this means Motion Capture has been triggered

TRIGGER_A: this means Actions have been triggered

Since you are getting notification/recording of actual humans as expected, this indicates that your settings are fine (which they seem to be from your screenshot) and that the AI is working correctly. So then the problem here must come down to how the consumer of the event stream is interpreting it and reporting the results. Since that's not our software, I can only speculate on what that problem could be - perhaps it is reporting a human detection whenever there is any kind of CLASSIFY event? Or perhaps when the HUMAN probability is any value over 0? Or perhaps even whenever there is a MOTION event?

benish

@Ben Thanks for the reply and insights into how the event stream works – I understand you have nothing to do with the Home Assistant plugin and I really appreciate you trying to help out here anyway!

I'm getting these false positives fairly frequently, but not often enough to suggest it's happening whenever there is any MOTION event. If fact, I can see regular motion events happening in Home Assistant that don't trigger the notification.

If you're curious, this is what the event stream data looks like in Home Assistant: https://go.benish.io/PJnOs7 Basically, I trigger a notification in Home Assistant when event_object changes to Human , and limit it to 1 notification every 30 seconds. There are different events happening for Human and Vehicle, so I don't think it's an issue of triggering on any kind of CLASSIFY event.

The false positives in Home Assistant mostly happen with one particular camera that has a shrub in the frame, which I think might be registering as a low probability human when there is wind. This would explain why SecuritySpy isn't recording these events (it's below the threshold). I would mask that area out, but it's also in an area where I want to make sure no one sneaks by. And I'm getting similar false positives on another camera that I think are shadows at certain times of day – and masking the suspected trouble area isn't feasible at all there.

Before your reply I cranked up the human threshold on this camera to 95 in the SecuritySpy settings, to see if that would help, but it seems to have had no effect on the false positives I'm getting from Home Assistant – which makes perfect sense now that you've explained how this works. The SecuritySpy for Home Assistant plugin has no threshold setting, and it seems SecuritySpy doesn't send the threshold values from the SecuritySpy preferences… so the plugin has no way of knowing what I set the Human threshold to, yeah?

So my best guess is that it's reporting a human when the HUMAN probability is any value over 0 – or perhaps some other value that I don't know and don't know how to adjust. @briis if you're around, could you provide any insight into how the SecuritySpy for Home Assistant plugin is handling the data from the SecuritySpy event stream? Is it registering a Human event_object for any value of HUMAN in the CLASSIFY event? If so, is there some way to set a threshold to reduce the number of false positives?

If it's possible to include the percentage probability from SecuritySpy as an attribute in the motion binary_sensor we could potentially trigger different automations in Home Assistant based on the probability of human? e.g. if >10 turn on a light… if >90 and in the middle of the night, start flashing lights and making noise, etc. But I think just being able to set a single threshold in the settings would hugely help decrease my false positives here.

benish

Now that I understand this better, I've opened an issue related to this on the SecuritySpy for Home Assistant GitHub repo, as that seems like the more appropriate place to deal with this.

@Ben sort of related: is there any way to see what the percentage probability score is (for Human and Vehicle) for a captured clip? As I fine tune my settings it'd be really helpful to know what the score is for a given clip.

jimmyjohnson

@benish Just dropping a note that I'd also find it helpful to be able to see the probability score for a given object detection so that I can up-titrate the threshold "just enough" after receiving a false-positive alert.

Ben

There is a way to see the AI classification results that is described in out blog post Optimising SecuritySpy’s AI Object Detection. Basically, you create a folder called "AI Predictions" in the SecuritySpy folder, and this gets populated with annotated image that shows the detection location and classification result. This is designed for temporary debugging purposes though, rather than as an ongoing procedure.

jimmyjohnson

Thanks. I saw that blog post but wasn't sure if it was still relevant. Thanks for the reply.

benish

@Ben Is there any way to use false positives to improve the predictions? e.g. tell the system that a particular prediction is wrong so it can learn? I sometimes get false positives with a very high score, such that it's not really feasible to adjust the threshold above that.

I get enough of these that I sometimes ignore the notifications because it's a false positive so often.

jimmyjohnson

Yeah. Just noticing this myself (thanks for the info on using the AI Predictions folder). I see that my chocolate lab can generate a 98% prediction score for human sometimes. (yes, he thinks he's a human so wonder if that plays into it?)

This is on a decent resolution camera (Unifi G4) at my backdoor where I really want to leverage object detection to alert me if a human comes through my back yard when my alarm is set to "Home Night" or "Away".....

Any other tips for these situations?

benish

For what it's worth, my cameras are decent resolution as well (3840 × 2160 / 8MP). Some of my false positives seem to be things like a shrub blowing in the wind, but getting a 90+ human score.

jimmyjohnson

I've tried bringing the threshold up to the high 90's and turned that setting for "length of detection" to 2 seconds. Still getting some false positives - especially when the dog stands up to try to open the door ( a human-looking behavior, I know).

Will keep testing.