How about adding gpt4 vision

killerdan56
killerdan56
in SecuritySpy

How bout adding a OpenAI vision option, with users needing their OpenAI api key to use it.

https://platform.openai.com/docs/guides/vision

so it can do something Like this

https://www.reddit.com/r/homeassistant/comments/1dgzuh7/extended_openai_image_query_is_next_level/

and maybe u can then generate a notification that describes what OpenAI vision saw.

Comments

  • killerdan56
    killerdan56

    and something like this as well

    https://www.reddit.com/r/frigate_nvr/comments/180r5v7/ai_summary_of_your_frigate_alerts_like_a_human/

  • jimmyjohnson
    jimmyjohnson

    I saw that reddit threat as well and thought the same thing. Would be amazing addition to SS.

  • Ben
    Ben

    Yes, large language models are becoming very powerful, and this kind of functionality on our radar for the future. However there are some issues to overcome:

    • SecuritySpy currently does all of its work locally on the Mac, which allows for fast processing and total privacy and control of users' data. Using these online services compromises privacy, since you're uploading potentially sensitive images (e.g. from cameras inside your house etc.) up to a server owned by a third party.
    • Sending images to an online API and getting back results is slow - not useful for realtime analysis, but could potentially be useful for some operations on captured footage.
    • Larger CCTV systems may have dozens or hundreds of cameras, recording frequently or constantly. Even if these LLMs were used in a limited fashion, this is still a lot of image data that has to be uploaded.
    • These APIs are not free; users would need their own paid subscriptions to these services.
    • AI algorithms for CCTV work best when trained on actual images from real CCTV installations. It's unclear how accurate these general-purpose LLMs would be, especially for nighttime/grainy images.

    Still, there is clearly huge promise in these new technologies. SecuritySpy's current AI model to classify humans/vehicles/animals is very accurate (98-99%), but of course has far less functionality than could potentially be offered by these LLMs. Ideally we'd like to get something running on-device - I think this would be appreciated by our users, many of whom decide to use SecuritySpy precisely because of its privacy advantages and because it provides users with full control over their data. Fortunately, Apple has provided us with amazingly fast chips with fantastic machine learning capabilities, although the large memory requirements of these models is a potential limiting factor.

Categories