Cameras consistently lag 10-20 seconds behind while viewing

punga

Issue: End user watches several cameras at her desk using SecuritySpy to connect to the server where SecuritySpy resides and records 12 various branded cameras. Two cameras (#5 and #12) in recent weeks have consistently been delayed from real time 10-20 seconds. These two cameras point to the entrance of the facility and when a vehicle enters, the user can see it real time 15 seconds before it appears in her views. This is a recent development as this configuration has been in place for nearly 2 years.

Configuration:
- SecuritySpy Server :
3.4.7 running on a MacMini Late 2012 0 2.3Ghz i7, 16GB RAM, 256GB SSD, running OS X 10.8.5 Server. Server also functions as FileMaker host for one database, CrashPlan storage server and file server. RAM is regularly 50% free and there is about 200GB free on start up SSD. SS regularly takes up 300-350% of processor load.

- Camera details:
12 cameras installed inside and outside the facility, most hardwired via gigabit, but a couple are wireless, including one of the two problem cameras. SecuritySpy is motion based recording (with no recompression) on to a dedicated 3TB partition of a RAID 5 G-Tech box via USB3. SS overlays time stamp info on all cameras, and there masks to block out road traffic from motion detection.
The two problem cameras:
#5 - Foscam FI8620 Wired IP (H.264) w/PTZ connected via switched gigabit.
#12 - Y-cam Bullet HD720 connected wirelessly via UniFi AP-Outdoor supporting 801.n wirelessly linked to a UniFi AP-LR also supporting 801.n.

- End User:
iMac 3.2 Ghz i5 8GB, 1TB running OS X 10.8.5 connected to gigabit network. Keeps SS running in the background to monitor gate and several other cameras onsite (7 total), connected to the server using MPEG-4 RTSP over HTTP method of connecting SS to SS. User logs out in the evening and logs back in each morning.
User typically has other operations going on in the foreground (MS Office, web browsing, etc.)

Troubleshooting steps taken:
- Restart wireless access points, cameras, SecuritySpy on server and even the server itself. In most cases, it seems to resolve the problem temporarily. But the issue continues to return, multiple times a day.
- changed configuration in server based SS Video Device settings to use RTSP over HTTP (was previously set to JPEG)
- The camera views appear in real time when the user logs into the individual web interfaces to monitor cameras, so the problem lies somewhere between the 2 copies of SS communicating as best I can tell. The user is at a remote site 3 states away, so it's hard for me to troubleshoot beyond making changes/restarting things and waiting for the user to report back.

Thoughts? Need more details?

Ben

This problem is caused when the video data that a camera produces is not consumed as fast as it is created, so it builds up in the camera's buffers, waiting to be sent. The better-designed cameras will lower their frame rate in this situation to avoid this problem, but not all cameras will do this.

The cause it either the network or the Mac being too slow to keep up. In your case it sounds like the Mac. If SecuritySpy is running at 300-350% this is near the Mac's full CPU capacity, and the other tasks performed by the Mac may be pushing it over the edge.

The main thing you can do to reduce CPU usage is to turn off SecuritySpy's text overlays. With these on, SecuritySpy needs to decompress and then recompress the incoming video data in order to save it to the movie files, even if you have the "No recompression" options checked.

Instead, enable text overlays in the cameras themselves. Also make sure that the cameras are set with an NTP (Network Time Protocol) server so that they keep perfect time (you can use time.apple.com).

Hope this helps.

punga

Re: turning off text overlays: Does this need to be done for all cameras or just problematic ones? I ask because each of the different brands of cameras has a slightly format/look for their text overlays, compared to having the consistency of the SS overlay.

Ben

The more processing you can shift to the cameras the better, so doing this for a few will help a bit, but doing this for all of them will help a lot. It doesn't necessarily have to be done with the problem cameras; your aim is to reduce the Mac's CPU usage in general, which should resolve the problem.

Recompression in software not only uses a lot of CPU but also degrades video quality, so if I were you I'd apply this to all cameras, and put up with the different format of the text overlays in order to get these significant advantages.

punga

I've been working on enabling all of the camera's built in time/text overlays. But I've run into an issue with the Axis M5014 models: Even though the text overlay is turned on in the camera's built in software, it does not show up in SS. I can see the text just above the top of the image in the camera's preview, but the text is cropped out (?) in SS and it's not visible.

Thoughts?

punga

Also, turning off the on screen display on 8 of 12 cameras (the non-Axis branded models) does not seem to have affected the CPU usage. Still hovering at about 375%. What did reduce the CPU percentage, yet not resolve the delayed viewed was disabling global compression settings. The consequences of disabling compression: filling up a 9TB drive in less than a week. So, compression is back on for the moment.

mbaumhof

i had also trouble with lagging only 4 cams on a MacPro 2014 model. With iStats i found that when the system is too hot the system task raised from 5-8 % up to 40 - 50 %. The SecuritySpy task shows also 300 %.
After cooling it down all was ok.

Michael

Ben

If after turning off SecuritySpy's compression (by using the "No recompression" option in the Video Device Settings window) you are getting very large file sizes, this probably means that the cameras are sending JPEG format video to SecuritySpy. JPEG video has a rather high data rate, so it's preferable to use either MPEG-4 or H.264 format video. So check the "Format" option in the Video Device Settings window and switch that to H.264 (if available) and/or log on to the camera using a web browser and change its streaming format to H.264. This gives you the best of both worlds: low CPU usage and efficient file sizes.

punga

Thanks for the tips on compression Ben.

Do you have an answer to my other question about the Axis cameras and their OSD being cropped out of the SS feed?

Ben

Sorry @punga I missed that question. SecuritySpy records the whole video frame, so this should include the camera's overlay text. If you right-click on the camera in SecuritySpy and select "Save as JPEG..." and then open the saved image file in Preview, do you see the overlay there?

punga

RE: timestamps:
Okay, very odd. When i was changing these setting yesterday, while watching the SS feed, the dates were not appearing. When I looked in just now to answer your questions, the info now appears in both SS and it's web interface of some, but not all four of the cameras. The timestamp does not show up in a still image from the cameras.

RE: compression:
My settings for the cameras that support H.264 are already set for that format in the Video Device Settings dialogue. I would post a screenshot, but there doesn't appear to be a way to do that in these forums.

RE: CPU usage:
Today SS's CPU usage is averaging below 300%, hovering around 250-275% as compared to yesterday after I changed settings for the overlays.

I will check in with the end user on the delay.

punga

Circling back to this: In the end, my solution was to configure the user's local copy of SS to talk to directly to the cameras instead to the central SS server. She needs to be able to things in real time and this was the best solution. We're still recording only on the central server.

Ben

Hi @punga thanks for reporting back, directing the viewing SecuritySpy directly to the cameras will certainly reduce the load on the server, so hopefully things are working a lot better for you now.