Where is the bottleneck?
Here's a fun game for the whole family-- It's called "Where is my bottleneck?" Only 4 FPS on 20 cameras, as viewed on the all-cameras display, command 0.
My system has evolved over the last 5 or 6 years. I started with a Mac Mini and I've graduated to the grand daddy of them all-- a 2010 12-core Mac Pro at 3.46 ghz. It was nice to put the Mac Mini on the shelf. Since I started with a Mac Mini and limited processing power, I thought it best to have jpeg cameras, mostly D-Link DCS-910 (640x480).
The system and network have grown to 16 D-Link DCS-910 (640x480) cameras, 2 DCS-2210, and 2 MPEG-4 firewire cameras running Remote Site. The cameras are each set to 20 fps and Security Spy is also set to record at 20 fps.
My home network is a hub-and-spoke Gigabit Ethernet-- Cat 6 cable between network switches. My Mac Pro uses one ethernet cable to the "main" Gigabit switch.
I record all 20 cameras continuously, so there is an enormous amount of jpeg data on the network. But the network doesn't seem terribly busy. Activity Monitor, Network reports about 6 MB/second. Other network activity is usually minimal-- it probably gets a little busier when streaming video on a TiVo Mini across the network from the main TiVo. (but i think that is highly compressed data). A change in network activity does not seem to change my fps rate.
Security Spy adds the date/time to the video, so it is processing the video. The Mac Pro's Activity Monitor reports 176 percent CPU-- it's barely touching the capacity of the machine. (this machine is a beast)
The video is stored on two spinning hard drives simultaneously (RAID 1 Mirroring). The first hard drive is inside the Mac Pro and the second is stored in my crawlspace (in case the computer is stolen) and is attached with an 8 foot Firewire 800 cable. Activity Monitor shows the data being written to the hard drives in spikes-- And as i watch it, the maximum data written per second is about 18 KB/sec, that's probably at the spike, but there is plenty of space between the spikes.
I'm only getting 4 FPS on the cameras. So where is the bottle neck? It certainly is not the CPU! Maybe it's the RAID disk writes? Or maybe the Gigabit ethernet can't really handle the jpegs? I hope I've been thorough in my description. Thanks for any help.
My system has evolved over the last 5 or 6 years. I started with a Mac Mini and I've graduated to the grand daddy of them all-- a 2010 12-core Mac Pro at 3.46 ghz. It was nice to put the Mac Mini on the shelf. Since I started with a Mac Mini and limited processing power, I thought it best to have jpeg cameras, mostly D-Link DCS-910 (640x480).
The system and network have grown to 16 D-Link DCS-910 (640x480) cameras, 2 DCS-2210, and 2 MPEG-4 firewire cameras running Remote Site. The cameras are each set to 20 fps and Security Spy is also set to record at 20 fps.
My home network is a hub-and-spoke Gigabit Ethernet-- Cat 6 cable between network switches. My Mac Pro uses one ethernet cable to the "main" Gigabit switch.
I record all 20 cameras continuously, so there is an enormous amount of jpeg data on the network. But the network doesn't seem terribly busy. Activity Monitor, Network reports about 6 MB/second. Other network activity is usually minimal-- it probably gets a little busier when streaming video on a TiVo Mini across the network from the main TiVo. (but i think that is highly compressed data). A change in network activity does not seem to change my fps rate.
Security Spy adds the date/time to the video, so it is processing the video. The Mac Pro's Activity Monitor reports 176 percent CPU-- it's barely touching the capacity of the machine. (this machine is a beast)
The video is stored on two spinning hard drives simultaneously (RAID 1 Mirroring). The first hard drive is inside the Mac Pro and the second is stored in my crawlspace (in case the computer is stolen) and is attached with an 8 foot Firewire 800 cable. Activity Monitor shows the data being written to the hard drives in spikes-- And as i watch it, the maximum data written per second is about 18 KB/sec, that's probably at the spike, but there is plenty of space between the spikes.
I'm only getting 4 FPS on the cameras. So where is the bottle neck? It certainly is not the CPU! Maybe it's the RAID disk writes? Or maybe the Gigabit ethernet can't really handle the jpegs? I hope I've been thorough in my description. Thanks for any help.
Comments
-
I have no idea, but I have some questions.
First off, do you mean 18MB/sec where you wrote 18KB/sec? Secondly, what do you mean with "I'm only getting 4 FPS on the cameras". Who reports the 4 FPS? The cameras or is that the video that's being stored? And how did you establish it's 4 FPS? What does ffmpeg -i [file] report, where [file] is one of the videos that's being recorded by SecuritySpy? -
Thanks for your efforts-- I'll try to clarify. In Activity Monitor under the Disk selection, it shows a maximum of 18KB/sec-- but as I watch, it varies-- and today I saw a brief maximum of 40KB/sec. And yes, it's KB. But those appear to be instantaneous readings, not average. Still, it's KB/sec not MB/sec, and the data seems to get thrown to the disk in spikes.
I see 4 FPS, as mentioned in the first paragraph, on the all-cameras display, command 0.
I had forgotten about the "command i" for the stored video-- it reports 7 fps (it varies from video to video, but that's the average). So the FPS reporting on the "all cameras" display does not necessarily reflect the FPS that is on the stored video.
For testing, I ran an additional copy of security spy on a 2008 Mac Pro, but wrote the data to an internal spinning disk, not Firewire 800, and without a RAID. During that test, I saw 10 FPS on most cameras and 30 FPS on the two Remote Site cameras!
It looks like I found the bottleneck-- writing all that jpeg data to disk is slowing the FPS on the 2010 Mac Pro-- especially since one of my disk drives is on a slower Firewire 800 cable. The RAID might have an effect too, maybe. I will eventually get a USB 3.0 card and run the storage drive on that vs. the Firewire 800.
Fun fact-- when i stared the additional copy of Security Spy on another computer, my main computer's version turned to Demo Mode. That's obviously to prevent two programs running on the network at the same time (since I only bought one copy). A nice precaution. It's a great program and one that should be protected, as Ben has.
Security Spy is teaching me a lot about video, cameras, network, storage... etc. Thank you in advance for any additional comments. -
Thanks for your detailed description of your setup, it sounds like you have put a lot of effort into building a fantastic video surveillance system!
I would be surprised if this problem were caused by the hard drives, but it's certainly possible - unusual drive setups such as RAIDs can be the source of problems. However 18 KB/s is hardly any data being written. And in fact it's certainly not consistent with 20 JPEG cameras being continuously captured. A typical 640x480 JPEG frame will be around 50 KB - for one frame. Multiply by 4fps and then by 20 cameras, and you should be seeing disk writing rates of around 4000 KB/s - or 4 MB/s. Are you sure the cameras are in Active mode in SecuritySpy and being written to the disk?
I would suggest that you don't need the RAID setup, but rather you could try reformatting the FireWire 800 drive and simply using that as the one and only drive that will be receiving the captured data from the cameras. It should be easily fast enough for continuous 4 MB/s transfers.
The Mac's CPU is not being taxed so that isn't the problem.
The network sounds extremely fast so that shouldn't be the problem, although sometimes if you have multiple switches in a chain they don't always play well together. Do you see any difference with the cameras that are connected to the same switch as the Mac, compared with the cameras that are connected less directly, to other switches?
Finally, newer versions of SecuritySpy have much more efficient network code than older versions, so you should make sure you are using the latest version of SecuritySpy. I see from our database that you are a 3.x license-holder, so simply download and install the new version, replacing your current version (no payment required).
Let us know how you get on! -
Thank you for the software update reminder. I have been running 3.4.7.
It’s obvious that despite building an effective camera system, I have forgotten settings I made a couple years ago. My inexpensive D-Link cameras deliver data to Security Spy as jpeg— no other options— BUT— I forgot about my compression settings for local capture.
Since I add date and time information, Security Spy is recoding and compressing. Under Compression Settings, Codec is set to MPEG-4. Quality is set to 50. Audio compression codec is AAC, Quality is set to 50. That’s why the low data rate written to disk.
A typical hour long video from the cameras is stored as a “.mov” file, MPEG-4 640x480, FPS 7, 250 MB size, data rate 550 kbit/sec.
As i adjust the video Quality Setting for local saving, the FPS rate as displayed on the All Cameras display changes. No matter what settings I choose, Security Spy CPU usage remains under 200%. My total CPU for the entire system usage usually shows 88% idle— Apparently there’s not much else going on except Security Spy. I’d be OK with Security Spy using more CPU to increase the frame rate, if I could force it. Any suggestion?
My old 2008 Mac Pro is running Security Spy (not storing the data) with the same settings and the All Cameras view shows about twice the frame rate (8-10). The 2008 is on a “spoke” of the network and the 2010 is on the “hub.” No cameras are attached directly to either switch. I think the network is handling the data fine. But why the difference in frame rate? There is no change in frame rate when I switch from Active to Passive mode on the 2010.
The network is, as i mentioned before, hub and spoke. Most spoke switches feed directly to my office switch (the hub), where my 2010 Mac Pro runs Security Spy. One switch, the detached garage, feeds to the "behind the tv" switch before reaching the hub. I didn't have enough Cat 6 direct-bury cable to reach all the way to my office. All cameras, no matter where the are on the network, show approximately the same frame rate.
My job takes me away from home for days at a time. During one trip, my Security hard drive crashed and I was left with no stored video at all. I replaced the failed drive with two hard drives that are optimized for video surveillance (WD Purple, I think)— They are set up as RAID 1. I set up the RAID in Apple’s Disk Utility. In theory, if one hard drive crashes, the other will keep recording and I can replace the bad one when I get home. The RAID might be slowing things down? Possibly. But no change when switching from Active to Passive mode in the All Cameras Display.
That’s all I can think of now— Please don’t interpret my frame rate discussion as a complaint in any way— Security Spy is amazing software— Certainly above my analytic ability, that's for sure. Again, much learning on my part. Thank you for additional comments. -
That makes sense about getting SecuritySpy to add a text overlay and compressing the streams as MPEG-4. Still, the data rate being written to disk is suspiciously low, however if the frame rate is the same with all the cameras in Passive mode, then we can absolutely rule out the disks being the problem, because in Passive mode the disk isn't being used at all.
As you say, a RAID 1, where the data is duplicated, sounds like a good failsafe setup in case one of the drives dies.
You mention in your last message that you have a second Mac Pro that is streaming video from all the cameras - I suspect that this may be the culprit. When you have multiple clients receiving video from a network camera, it will impact on that camera's performance. Your DCS-910 cameras are capable of sending 640x480 video at a maximum rate of 15fps. I believe this 15fps will be shared between all connected clients. So if you are getting 8-10fps on one Mac and 4-5fps on the other Mac, that's probably about the maximum that the cameras can supply.
If you want to use the 2008 Mac Pro as a viewing station, perhaps it would be better for it to obtain the video streams from the main 2010 Mac Pro rather than from the cameras themselves, so that each camera only has one connected client (the main Mac). This setup will put a small amount of extra load on the main Mac, but it does have a lot of extra capacity spare so this shouldn't be a problem. -
I took the data rate being written to the disk from Activity Monitor, so it might not be showing an average data rate, just an instantaneous one--- so if it doesn't capture the data rate right at the peak, it might not be accurate. Like I said, data looks like it's thrown to the disk in spikes.
When I was adjusting settings and preferences inside Security Spy, the response seemed laggy-- a delay between clicking and the pull down menu appearing. I know it's not the CPU. BUT the SecuritySpy User Manual warns about running the All Cameras window (which I love). I adjusted the frame rate in Preferences to "display at half rate" and the main CPU usage went up. Then I closed the All Cameras window and the CPU went up again-- now it is around 300%. And the stored video frame rate is about 13 fps. With 12 processors, 1200% would be fully used. I'd be comfortable with SecuritySpy using more than 300%, if it could.
After all the searching, I believe the bottleneck is my video card-- although upgraded (NVIDIA GeForce GTX 680, 2 Gigs of Video RAM), it is not strong enough to run my displays and the All Cameras window without slowing the total data throughput. I will try to run SecuritySpy without the All Cameras display up when I'm not at the computer, but gosh it's a great display when I'm here.
Thank you for the suggestion about the 2008 Mac Pro. That's my old machine and it's just here until I'm sure my (craigslist purchase) 2010 is stable and running reliably, then I'll sell the 2008. I was using it for viewing and testing. Your idea to broadcast to another computer is a good one. The 2010 has enough CPU to do it easily.
The mystery is solved-- video card in combination with the All Cameras window. Thank you for your help and I'd appreciate your comments. -
This is very interesting - it certainly sounds like you have found the culprit. Actually the latest beta version of SecuritySpy includes a change to the video display code that may impact upon this situation, so please try it out and let me know if showing the "All cameras" window has the same impact or not.
SecuritySpy uses OpenGL to display video to the screen, and this is all asynchronous: once SecuritySpy makes a request to display a video frame, it doesn't know exactly when this request will be actioned by the video hardware. The video hardware may be busy doing something else, so there may be a short delay.
Previous versions of SecuritySpy used "shared memory" mode, where the memory that holds the video frame is shared between the app and OpenGL. This is fast, but due to the asynchronous nature of OpenGL, sometimes a frame would be accessed by OpenGL while that same block of memory was being used by SecuritySpy to prepare the next frame, and this can lead to visual anomalies in the video display.
So, a few versions ago, we switched SecuritySpy to the "cached memory" option, where upon every request to display a frame, OpenGL copies the frame to its video RAM. This eliminates any visual anomalies, but is slower and uses a lot more VRAM. We got a few reports that this was causing problems (slow performance and crashing with high resolutions and not enough VRAM), so in the beta version we've switched it back to the shared memory mode. So it will be interesting to see if this is the cause of your problems too. -
I swapped a little hardware-- Added a USB 3.0 card to feed the second half of the RAID a little quicker. (did not affect anything) And more importantly, I'm borrowing a NVIDIA Quadro K5000 with 4GB VRAM. (did not affect anything either)
Oh and since I didn't mention it earlier, my system has 16 Gig RAM, and Activity Monitor shows almost no memory pressure-- it's low and green.
SecuritySpy 3.4.7 with the fancy video card-- All cameras showing, 170% CPU and shows 4 fps on most cameras. All Cameras window closed, nearly 300% CPU.
SecuritySpy 3.4.8b9 with the fancy video card-- All cameras showing, 290% CPU and shows 15 fps on most cameras. All Camera window closed, 290% CPU, no change.
Personally, I prefer the beta version and how it handles the All Cameras window. It keeps the CPU up and maintains a nice frame rate on the All Cameras window.
This Mac Pro is a beast-- even more so with the Quadro Video card. If the Quadro can't keep up with video, etc., I don't know what could.
I have screen shots and I'd be glad to send them your way, if you'd like. I'm sure you have my email address in your database of purchases. Thanks again-- this has become very interesting! -
Thanks for the results, yes I agree, I think we'll keep the shared memory mode with the video display in order to maximise performance. Any display anomalies will be mild and transient and won't affect the captured footage anyway. With SecuritySpy, we consider high performance to be extremely important. We'll release the full 3.4.8 version next week, as it has some other improvements too.
-
You really need to try this without the RAID setup. The system is going to buffer that data differently and you won't get accurate reporting from Activity Monitor. `top`and `iostat` may provide better data (try `iostat 1`). If you can determine and concede that the problem is with the RAID, we can explore other options for backing up your footage.
-
I believe we eliminated the drive setup as an issue when I switched between Active Mode (write data to disk) and Passive Mode (no tats written to disk). There was no change in FPS seen on the All Cameras window.
The CPU / FPS did, in fact, change when I closed the All Cameras window.
I am currently running the beta version (with shared memory mode for video), as Ben suggested-- the CPU and FPS are high with the beta, but not with 3.4.7.
I'll definitely remove the RAID setup when I get a chance-- just to see if anything changes. Thanks for the input.
-
Also, to backup your data, you may want to consider dropbox. It's what I use.
https://blogs.dropbox.com/tech/2014/07/streaming-file-synchronization/ -
This morning I rebooted my system with one half of the MacPro's Security hard drive RAID attached-- the one internal to the MacPro. The system told me that half of the RAID was missing, then continued recording the cameras at exactly the same rate as with both drives attached and working. A couple hours later, I reattached the external half of the RAID (USB 3), and it updated, making it identical to the one that had been working all along-- and simultaneously recorded the cameras at the same rate. Apparently there is sufficient bandwidth to handle both halves of the RAID live camera feed and restore missing files at the same time!
Dropbox is a great way to keep an off-site backup of data-- especially security camera data. I've attempted using dropbox in the past, but I did not have sufficient upload bandwidth to store all 20 cameras' data in a timely manner. It would be worth my time to revisit Dropbox to see if I can make it work for me now.
My goal is to capture a would-be burglar on video as s/he arrives at my house, breaks in and searches the house for things to steal. The burglar will likely unplug a network switch or two (removing cameras from the feed) and try to remove my MacPro. The MacPro will be, at a minimum, unplugged and the security camera video will stop. The computer is locked closed and attached to my desk using a Noble Security Lock. Of course a good bolt cutter would cut the cable.
Since the MacPro will be unplugged and possibly stolen, it's important to maintain a second hard drive in my crawl space with duplicate security video.
Of course this may seem quite paranoid, but I live near Seattle and two families I know have been burglarized in the last couple years. One right in the city and one in the suburbs. One burglar was caught with stolen items, but, since they could not prove he had been in the house, was charged with "possession" and not "burglary." That's where SecuritySpy comes in.
My security camera project has been (what I'd call) "healthy paranoia" and it's been an excellent thinking and learning exercise-- Plus it's been fun watching the dogs. Thank you again for all the input and suggestions. -
It definitely pays off to have a certain amount of paranoia - if the burglars made off with your Mac Pro (which is something they might very well try to steal), and you had only one copy of the security footage on the Mac Pro's hard drive, this would be rather upsetting. Your RAID solution sounds great - especially impressive that it automatically catches up after disconnecting and reconnecting the drive.
-
I'm very pleased with the framerate on the beta version (b9)-- but it has had consequences. I just went on a week trip and came back to just 5 days of video on my 3 TB drive.
Increased frame rate means larger hourly videos. Yep, I understand that-- and it was interesting to see the results. To have more than five days of saved video, I'll need to-- 1. get a bigger set of drives. or 2. decrease the frame rate. or 3. set up some motion detection.
Fascinating how one change affects other limits. Wack-a-mole comes to mind.
