New Human/Vehicle Classifier - Please Test
Hello all, we have been working on a new and improved AI human/vehicle classifier and it would be very useful to get testing feedback from users so that we can make sure this is working correctly on all the different Mac hardware out there.
In all 5.x SecuritySpy versions until now, we have separate binary classifiers - one each for humans and vehicles, but this means that to detect both, a frame has to be processed through each classifier in turn. We have now trained a new multi-label classifier, so that only one pass through the new classifier is required to predict both humans and vehicles.
In addition, we have made major optimisations that significantly improve model performance (on some Macs, the new classifier runs >5x faster than each of the old ones).
Finally, by expanding our training dataset, and more careful training optimisation, we have increased the accuracy. The binary accuracy of the new classifier is 97% (i.e. both humans and vehicles are correctly predicted in 97% of frames sent to the classifier). For humans only, the accuracy is above 97%, and for vehicles only, it is above 98%.
We have tested this on all the machines we have here and it works great, but due to the fact that there is a wide variety of different Mac hardware out there, with various methods of acceleration for machine learning models (different GPUs models and now different M1 chips with neural engines), it would be very useful for us to get testing feedback from users, before we release this as an official update. So, please do the following:
- On the version of SecuritySpy you are currently running, click the SecuritySpy menu, and from the Debug submenu, select the Run Classifier Test option. This will display classification results from a few images. Please report 1. whether these are all correct, 2. if any errors are reported, and 3. the reported classification time in milliseconds.
- Then, download and install the latest beta version of SecuritySpy (currently 5.4b1), and then re-run the classifier test. Please either post a screenshot of the results, or report all the same items as above, plus the exact model of your Mac and its GPU.
- Continue using the beta with your normal settings, and report any problems you see with the ongoing human/vehicle detection.
I'm looking forward to seeing the results! To get the ball rolling, here are the results from my iMac (classification time with the old models was ~50ms):
Many thanks to anyone who can help with this!
Comments
-
Here's my results with the current beta and my M1 Mac Mini (16gig ram).
-
M1 Mac Mini with 8GB (not enough) RAM:
-
It's definitely faster.
Before:
After:
Although my results seems pretty slow compared to other results here.
-
These are the specs of my mac mini. Is my classify time normal?
-
Here are my release and beta screen shots -
-
Thank you everyone for the testing! This is very useful.
@emergent your classify time of 38ms perfectly good. And it's great to see that it's much faster with the new algorithm. On Intel Macs, it all depends on whether the Mac can use the GPU for hardware acceleration (which does seem to be the case for yours, otherwise it would be much slower), and then it depends on the specific model of GPU, as they vary quite a lot in terms of their capabilities. The M1, with its multi-core dedicated Neural Engine, just blows away all previous Macs in terms of neural network performance.
-
In that case I hope they release some Mac minis with the M1 Max soon.
-
Got anything more difficult?
-
does this mean you might release an animal detector soon?
-
Hi @jimmyjohnson we are in the process of training a new algorithm for this. Whether we release it or not depends on what accuracy we end up with, so I can't promise anything yet, but I'm hopeful at this point.
-
Hello. In this beta, preference of the web server, its possible to restore the name of the server (the one displayed in iOS app) ? Thanks!
-
Hi @pgh that setting is now called "System name" and it has been moved to the General settings. Originally this was only used for the web server, hence it was in the web server settings, but now it's used elsewhere (email motion notifications, error reports, daily stats email) so this now makes more sense under the general settings.
-
Do you think it'll be possible to one day detect people or vehicle movement direction using AI? e.g. Human detection moving top to bottom (towards the house or away from the house) vs human detection moving left to right or right to left (across the end of the drive way). We need to know when courier drivers are walking up our drive way, but we don't need to know when people walk across it at the end of the driveway.
-
Thank you for everyone who tested this. Along with our own testing, I think we have enough feedback to conclude this is all working very well. We'll release an official 5.4 update with this improvement (and others) soon.
