Low-res stream triggering hi-res actions - resource saver?
This might be something that is discussed elsewhere, but a quick keyword search doesn't find anything easily referenced.
I have cameras which have "high-resolution" and "low-resolution" streams. If I activate motion detection on all my "high-resolution" streams, I tend to overwhelm the system. I don't know if it's CPU or GPU (my CPU numbers don't seem to climb above 140%) but when I have a lot of active cameras I notice that capture files get really choppy, and my thumbnail views also slow down significantly.
What I'd like to do is use my low-resolution streams for detecting motion, and then as an "Action", trigger a high-resolution stream camera (the same network device, but a different stream) to start recording at the higher frame rate/resolution/quality.
I'm not sure if this would help my problem, since I don't know where the bottleneck is with SS. If I have two streams from the same camera, but only look at motion on the low-res version, does that increase system performance? If I have a network device that has no motion events configured and no continuous capture configured, and it is just in "waiting" state standing by for an "Action" that tells it to start recording, does that cause significant resource drain? I'm looking to have two TCP/RTMP sessions going here - the low res one that gets examined and uses resources, and the high-res one that is essentially "dumped" to /dev/null until the low-res action occurs and triggers a recording event on the high-res stream, at which point recording starts on the already-running high-res video session. (it wouldn't actually be zero resources, since memory would be used for pre-event recording on the high-res stream, but hopefully that is minimal?)
I did already try doing this with "Trigger Other Camera" as an action, but I have to have Motion events configured on the high-resolution stream in order for a capture file to be created (actually, that doesn't work either but I don't want to stack issues here.) I fear that having Motion events active on the high-res device defeats the purpose - I'm already "burning" resources on the high-resolution stream to do motion-detection so having the low-res stream as the trigger source seems duplicative and un-necessary.
JT
Comments
The main thing to consider here is that it's not motion detection per se that causes resource usage - this is a very quick algorithm - what causes the biggest resource using is the decoding of the incoming streams, which is required for motion detection but also required for other tasks.
The incoming video needs to be decoded for the following functions: motion detection, live video display, image capture (including uploading/emailing images), live viewing via the web interface, live viewing via the iOS app.
It's possible to turn off decoding when not required, via the option in the General Preferences called "Decompress incoming video frames only when required". This option is off by default, as it provides a better user experience (e.g. when a video window is suddenly opened, a frame is available immediately for display, rather than having to wait until decoding to start again, which could take several seconds).
So, you could in theory save processing time by using low-res substreams, which trigger the main streams, but only if the main streams aren't normally being decoded for other reasons. If the main streams are being decoded for other reasons (e.g. for display in video windows), then adding the substreams will increase your overall resource usage, rather than reduce it.
Overall, I would not recommend the use of substreams in this way. It's much better to use a single high-quality stream for each camera, making sure the Mac hardware you are running on is capable of processing this (see our System Requirements Calculator). If your Mac is overloaded, reduce camera frame rates until it runs smoothly.
One other thing to note: video decoding is done in specialist hardware on all modern Macs, so it won't show up in the CPU usage metrics reported by Activity Monitor. But the decoding hardware can still become overloaded, leading to poor performance. Check the log file (File menu > Open Log) for any messages related to "decoder overload", which will warn you of this situation.
Currently, it appears that this is not merely sub-optimal, but impossible as I describe. The second HD "main" stream camera must have motion detection enabled in order for it to be triggered by an action from another low-resolution stream, which means even if "Decompress incoming video frames only when required" is checked on the HD stream. Am I misunderstanding this? I wasn't able to get it working otherwise - sending a trigger action request to another camera that is not already configured for motion detection doesn't seem to have any effect, and if motion detection is configured on the HD main stream then it's always decoding, thus defeating the goal of reduced processing load.
To answer your question posed in the second paragraph: No, ideally the main (HD) streams would not be decoded for any reason until a trigger event from a low-resolution stream caused them to be recorded (decoded.) They would be lacking any decoding until needed, though I suspect it might be required that the stream is active so there isn't the latency of the TCP/UDP session startup when recording would start, and also so that the pre-capture stream data could be somewhere in memory for re-wind/key frame collection and decoding once the motion detection trigger was received.
I agree that the ideal way to do this would be to get more horsepower to throw at the problem and take only the high-load streams, but it does seem like a fairly reasonable way to extend SS's utility (possibly quite extensively) without spending more on hardware, making it a more appealing solution given the high costs of larger Apple hardware these days as compared to alternatives. Low-bandwidth streams are useful enough for motion detection, and also useful enough for thumbnail views. The important events that need to be forensically captured would all be using the main streams, which would be much higher resolution, but the larger framesize (or faster FPS, or higher quality, or whatever) decoding load would be applied to the system only for the duration of the observed motion event.
To make it more in economic terms: this method possibly would allow customers to have more camera licenses on a single SecuritySpy server, rather than deferring equipment and SS license upgrades until larger hardware can be budgeted. I know I've seen a few threads (and have myself had hardware constraints) that mention this type of economic equation, which then in turn has a reflection on how many licenses you are able to sell. Not only could this be a 'savings' option for hardware, but for those who have limited network bandwidth (WAN or WiFi cameras) this could also be a winning strategy if there was an option to not even start the stream until the action triggered the motion detection event. (I would not want this bandwidth-savings option in my network and would want the streams always being pulled in, but some might find it extremely useful.)
When one camera triggers another, you don't have to have any triggers enabled for the second camera; you can turn motion detection off for the second camera. What you must have set is that the first camera is set to trigger the second under Preferences > Cameras > Actions, and that Actions mode for the first camera is armed (because triggering another camera is classed as an Action), and that Motion-Capture mode for the second camera is armed. I have just tested this, and it works as expected. What you should see is that when there is motion in the first camera, the "A" (Actions) indicator turns red for the first camera, and the "M" (Motion Capture) indicator for the second camera then also turns red.
You are certainly right that with the correct setup and careful usage (e.g. not leaving the All Cameras window open, requiring all cameras to be decoded), then this functionality can extend the use of older Mac hardware and allow the use of more cameras than would otherwise be possible. I hope with the above description you are able to get this working.
I got the trigger to finally work but the action->motion detection trigger wasn't functional until I restarted SS, which I think was where I was having a problem. Minor bug somewhere...
So now that I have the actions working and can see how this "works" I'll still go with my original premise: if I have to activate Motion Detection on the "target" camera, then this doesn't seem to me to save any CPU. I have "Enable Video Motion Detection" un-checked for the HD camera device. I do also have "Decompress incoming video frames only when required" selected, but of course if the "Motion Detection" active on the HD stream (I think) means it's doing decompression so I'm not sure where to go here...
The HD (3.7MP) camera device shows 4.5% CPU use, while the low-resolution stream device (.4MP) shows 2.1% CPU use. My hope would be that the HD camera device shows 0% (or very small) CPU use until such time as it is in Motion Detect triggered mode as a result of the action of the low-resolution camera. The "Decoder Pressure" bar sometimes is much higher for the HD stream the the LR stream, sometimes the opposite - I'm not sure how to interpret that, but the fact that there is "Decoder Pressure" happening at all during quiescent times tells me that everything is being decoded all the time.
...and sure enough, looking at the "Decoder Pressure" graph of the HD versus the LR cameras shows that the HD camera is much, much higher (between 20-40) than the LR camera (1-4) on that graph, with the settings you describe. Even going so far as to deactivate motion capture (disarmed) there is still pressure on the HD device. This is contradictory to what I'd expect - is this behavior normal? A device with no motion capture, no continuous capture, no actions, preferences window closed, no web connections - it's still exerting decoding pressure for what should be a dormant camera.
From your description, I think this is working as expected, however a few complexities mean that it's difficult to determine from the UI whether this is actually the case. Therefore, in the latest beta version of SecuritySpy (5.5.4b1), we have made the following changes to make this clearer:
Please let me know your feedback when you've had a chance to test.
Thanks for the beta fixes! Still seem to be some unexpected results, though.
My three devices (all H.265):
TestCam1 - low-resolution (stream 2) feed, no audio, active for motion detection, with 24x7 continuous capture at .2/25/daily/no upload, no Motion Capture, and two actions - a) Trigger Motion Detection on TestCam1-photo, and b) Trigger Motion Detection on TestCam1-HD
TestCam1-Photo - high resolution (stream 1) of the same camera as TestCam1. No audio, no motion trigger, but it does have image capture (jpg) upon motion detection event. The only way this can "fire" is if it's triggered by an Action from TestCam1.
TestCam1-HD. high-resolution (stream 1) of the same camera as TestCam1. No audio, no motion trigger, but it does have "Capture a movie" in response to Motion Detection events. The only way this can "fire" is if it's triggered by an Action from TestCam1.
Here's what I see with my three devices now:
I am presuming that "Trigger Motion Event" and an actual motion event have the same behavior path. (My use of the manual motion detection is frustrating, since I am testing this on a test system/camera that I chose because I thought the camera would have lots of activity with blowing branches, but today is entirely windless for the first time in weeks and I'm too busy to set up some other camera where I can wave at it. <shrug>)
PS: the dashboard graphs are really useful for helping sort this kind of thing out - thanks for those.
JT
Update: the decode pressure for the low-res stream device (TestCam1) and the decode pressure for the high-res stream device (TestCam1-HD) are often identical. (blue is TestCam1, purple is TestCam1-HD)
Hi JT I'll reply to each of your points in turn:
By the way, an easy way to generate a motion trigger in the source camera (TestCam1) is to right-click on it and select "Trigger Motion Detection". You won't have to wait for blowing branches, or have the camera within waving distance!
1) OK.
2) TestCam1-HD shows "HW" no matter what I set things to (or it quickly moves to "HW" after a brief moment of "...") while TestCam1-photo shows "HW" during motion capture as triggered by action on TestCam1 and then goes back to "..." when the Action event is done (this is the expected behavior.)
3) The red "M" does not go back to orange after the post-capture time for TestCam1-HD after it triggers once. It stays red. Double-checking the "Movie Capture" settings show 15 frame rate, 4 pre-capture, and 7 post-capture.
4) No web connections, no video windows, no audio, no preferences window open, no motion detection. I've restarted SS and even turned off the web browser entirely and restarted - same result. The "M" stays red. Note that the "M" goes orange on TestCam1 (low-res) device as it should.
5) OK
6) OK
I have been generating all of my events with "Trigger Motion Detection" in this way manually - that's what I thought was ironic, I chose the camera that was triggering most often, and that day the wind stopped.
Happy to give more debugging on this in mid-January when I surface from my end-of-year madness. Just let me know what I should provide other than the usual debug files, if anything. I'll also pare the system down so that only these cameras exist/are active if that might help.
What you describe with the M staying red does not sound right and indicates a possible bug with this mechanism of one camera triggering another. We'll be able to look into this in more detail the first week of January and will report back with what we find.
Just following up on this - in the recent v5.5.4 release, we made some changes to the way this triggering works, so I'm hoping that this will fix all problems you have mentioned above and that everything will work as expected now. You can update via the "Check For Update" option in the SecuritySpy menu. Please let me know once you've had a chance to test.
Still not quite there with the updated version. In that same setup as I describe above (low-res stream device that has motion detection and actions, second device that takes a photo based on action trigger from low-res stream, third device that connects to high-res stream and records motion detection only based on trigger from low-res stream):
The low-res stream triggers during motion events, the photo stream shows a red "M" during action/motion events on the low-res stream but pictures are not stored, and the high-res stream gets the "M" to turn red, but then it never turns off and no videos are stored.
Also: Triggering motion detection with the right-click menu on the low-res stream is now ineffective. I have to walk outside to trigger the low-res camera.
Also: the high-res stream seems to be decoding 100% of the time. The "HW" notice is beside the camera pressure indicator.
I even simplified to just low-res, high-res streams. Same results.
I see that motion detection is happening on the HD camera - this will require decoding. To prevent the HD camera from decoding, check the following settings:
Preferences > General > Suspend motion detection when not needed: ON
Preferences > General > Suspend video decoding when not needed: ON
Preferences > Cameras > TestCam1-HD > Triggers > Motion Capture Triggers > Video trigger: NONE
Preferences > Cameras > TestCam1-HD > Triggers > Actions Triggers > Video trigger: NONE
Preferences > Cameras > TestCam1-HD > Triggers > Movie search metadata: both OFF
As for the other issues with lack of recording and the M indicator staying red, I just can't reproduce this or see how this could be possible, so this is a mystery at the moment. Make sure under Preferences > Cameras > Motion Capture for the HD camera that you have either the movie or image capture option enabled. With either enabled, when the M indicator turns red, there must be some files being saved. Could you please double-click this?
I had the "Preferences > Cameras > TestCam1-HD > Triggers > Movie search metadata" set to "On" so I set to "Off" but everything else was already configured as you noted. Sadly, no changes. "M" goes red, stays red. Files apparently are getting recorded, but they don't stop until the camera loses connectivity (this is a WiFi camera that loses link every few hours- I should switch to a wired camera, but I don't see how that would have an impact on the underlying problem.)
<bump>
Hi @jtodd would you be willing to give us temporary TeamViewer access to this Mac to run some tests? I think this would be the best way to get to the bottom of this. I have checked the code thoroughly and run tests here, and cannot reproduce this problem that you are describing with the motion-capture recording getting stuck, so I'm currently at a loss to explain what you are seeing (I assume you're on the latest v5.5.7 release now?). Screen Control via SecuritySpy's web interface would also work if that's already set up. Please email us to arrange a session. Thanks.
Yep, can do that in about a week when I'm back at my desk.
Great, I will look out for your email!
A bit more hackery today before fixing this the "right" way with Ben... I wrote a short shell script that would do sort of what I want. It's sub-optimal but... it does work. I've only applied it to four or five cameras (all 2k/4k) - you can see in the graphs where I applied the script and I have four or five more to go - and my CPU and GPU numbers have dropped significantly. This editor leaves a lot to be desired for pasting shell script... (disclaimer: I actually wrote this quite a while ago and was very sleep deprived at the time so it may have other flaws that I haven't noticed)
#!/bin/bash
# usage: ./secondary-activate <camera-number>
#
# jtodd@loligo.com
# v0.1 2023-05-09
#
# SecuritySpy (www.bensoftware.com) video surveillance add-on script
#
# Goal: to reduce the CPU and GPU usage on very busy or older gear
# that can’t keep up with a large number of high-resolution
# cameras. Use just the low-resolution streams for examination
# and then turn the high-res streams on temporarily when
# something interesting happens.
#
# This activates Motion Detection 24/7 for a camera, triggers a motion event,
# then turns off motion detection. To be used as an “Action” event by low-res
# camera streams which can then “wake up” the higher-resolution camera for
# recording of the event. This removes the load from the CPU from parsing
# high resolution camera data, since low-res cameras are usually sufficient
# to trigger on motion events.
#
# Just create clones of your high-res cameras, change the stream from ‘1’ to
# ‘2’ (which is almost always the low-res stream) and add this action into the
# new low-res camera as a shell command, using the high-res camera number
# as the argument. So “/usr/local/bin/secondary-activate 12” is the command
# you’d use on your low-res camera, if your identical high-res camera is
# number 12.
#
# Then turn off scheduled or 24/7 status for continuous/motion/actions on
# the high-res camera but leave everything else the same.
#
# I would suggest the “Action delay” be set to zero and “Action rest time”
# set to 1 on the the low-res camera.
#
# Make sure to have “suspend motion detection when not needed” and
# “suspend video decoding when not needed” checked in the General preference.
#
#
# This script is very dumb; it could be the case that it gets “stacked” in
# a busy environment, where it’s running several times based on events detected
# on the low-resolution camera, which will cause unexpected results or lost
# events.
#
#
# Get the authid by using the user/pass combination you created in the “Web”
# preferences. Run this command to get the authid: “echo -n user:password | base64”
#
authid="aeRvZ3Q6Zm93XmPyMBIz"
# Wake up the high-res camera specified - set Motion Detection status to 24/7
#
curl -X GET "http://127.0.0.1:8000/setSchedule?cameraNum=$1&mode=M&schedule=1&auth=$authid"
# Delay a moment to let this take effect (needed? Not sure.)
#
sleep .5
# Trigger a motion detection event on the high-res camera we just activated
#
curl -X GET "http://127.0.0.1:8000/triggermd?cameraNum=$1&auth=$authid"
# Give a healthy wait before we shut the camera back down. I should probably
# be more clever here and check to see if there are any motion events happening
# on the camera before turn off the Motion Detection schedule, so we don’t shut
# the camera off during interesting things.
#
sleep 30
# Turn off Motion Detection schedule to be “None” instead of 24/7
#
curl -X GET "http://127.0.0.1:8000/setSchedule?cameraNum=$1&mode=M&schedule=0&auth=$authid"
This script above is not a "bandwidth" savings tool - it is a processor savings tool. It will actually increase bandwidth, since each camera will send both a low-resolution and high resolution stream of data to the SS server. However, only the low-res one will be decoded and examined for motion events. If you want to use it for a bandwidth savings tool, you could possibly re-write it to actually activate/de-activate the whole high-resolution camera entirely, which would (I assume) stop the video stream from even being fetched. It would take some time for the high-res stream to be activated, connect, get a key frame, etc. but for people who are using SS over really poor low-bandwidth links or using bandwidth that is very expensive (satellite) then maybe it would be a reasonable solution for cutting back on bitrates by 80-90%. Just modify the script to use the "camEnabledCheck" value described in the API. (https://www.bensoftware.com/securityspy/web-server-spec.html)
Just an update on my script - this is working surprisingly well. I have converted all my high-resolution cameras to use this model, and now on my 12 core Intel Mac Mini, six of the cores are entirely idle, and the six cores that are active are averaging only around 35% usage, and I suspect that is mostly due to six older low-res cameras which are ancient and still using software decoding for JPEG streams. The GPU is at around 20%. This is with 38 total streams. Decode pressure is almost none. (All my streams are VBR, if you're wondering why the data rates are a bit odd.)
Note that many of the "Dec" column entries are "..." meaning (I think) that there is no decoding happening. These are the high-resolution streams. Each high-resolution stream has a corresponding low-resolution stream just below it from the same camera. All the streams (high and low resolution) are active and being pulled in by SS, but only the low-resolution streams are being decoded and examined. When a motion detection event is triggered by the low-resolution stream, the script (as an action) is fired, causing motion detection to be enabled on the high-resolution stream, and a motion detection even to be forced on the high-resolution stream. Then the high-resolution stream chews up CPU while being decoded. After 30 seconds, motion detection is turned off on the high-resolution stream, and we go back to where we were.
This has led to a drastic reduction in my CPU/GPU usage.
Ben and I worked on my underlying concern some time ago, which was not script-based, but used Actions from low-res stream to turn on a high-res stream. This worked quite well, and removed the need for this script. The problem had been that all my cameras had "Recompression" turned on, which meant that they never were quiet - they were always using "HW" resources.
So I found out today why I had recompression turned on, and I'm back in the same boat... unless a patch can be made. The reason I had recompression enabled was that I have Continuous Capture turned on, but my frame rate is less than 1 frame a second (it's set to .2 frames a second, but sometimes I set it down to .1 frame per second.) This is because storing continuous streams at real-time speed is an ENORMOUS disk hog, and I really only need one frame every 5 or 10 seconds.
I discovered this today when I went to look at one of my continuous capture movies. Despite having a frame rate of .2, they were "real time" 10 frames a second, for the whole day. I didn't see the little triangle/exclamation mark beside the frame rate text block in the Continuous Capture area.
So I may have to fall back to my crude script method. But it does seem that there is a
So my goals is to reduce CPU usage and disk usage while getting:
It sounds like this would be possible, but this combination just isn't considered at the moment. Turning on lower-than-realtime frame rates requires recompression to the degree that it swamps my processor on the stream processing. The ugly script method I use above does in fact do the "right thing" but in a totally wrong way (where "wrong" means "ugly hack")
So that explains why you had recompression enabled! Yes, recompression is required if you want to change the frame rate of the incoming stream, and without it, 10fps continuous video recording could indeed use a significant amount of storage space.
With recompression enabled and continuous-capture armed and recording, SecuritySpy does need to decode the incoming video all the time, so that decoded frames are available to be re-encoded for recording. There is no way around this.
A few possible solutions: