MacPro slower than iMac
I am having a strange problem with lagging video and low frame rates on my MacPro but not my iMac.
I have been using SecuritySpy very successfully at my office building since version 1 and am now using version 3.4.10. I have added and upgraded cameras so that I now have 25 2 MP IP cameras: AvertX HD30, Samsung SNV-6013, and Samsung SNV-7084. All are H.264 formatted streams at 8 fps to Gb POE ports on a Netgear GS753TP switch through a separate private network. The switch is directly connected to my Macs.
All cameras are configured uniformly on SecuritySpy with Format: H.264 RSTP, No recompression, Text overlay disabled, Motion detection sensitivity 50, One movie per day, Frame rate: 8. The Capture destination is a 6TB RAID connected to the Mac by Thunderbolt. Both Macs are running OS 10.11.4 with SecuritySpy 3.4.10 configured identically and do not serve SecuritySpy to other Macs.
The system runs perfectly when I have everything connected to my iMac with a 4 core 4 GHz Intel Core i7 CPU and 32 GB of RAM except that I occasionally need to optimize its RAM. The iMac is also running various network monitoring and database apps at the same time.
I thought it would be best to have a new Mac dedicated to SecuritySpy which would allow further improvements so I purchased a new MacPro with 12 cores and 64 GB of RAM. All apps run normally on the MacPro except SecuritySpy. On the MacPro, SecuritySpy always starts up with all cameras showing 8 fps as expected with smooth motion in real time. After a few minutes, various cameras start showing fluctuating frame rates of 3-5, motion that lags as much as 10 seconds, and sometimes freezes. The captured video shows the major skips. Eventually the frame rate drops for all cameras, although it continues to vary, and does not differ by camera brand or model.
Troubleshooting:
- I ran hardware tests on the MacPro which showed no problems.
- I talked to Apple who said the problem is software related and confirmed that my MacPro graphics card is not defective.
- I reformated the MacPro solid state drive and installed a fresh OS 10.11.4 and a fresh SecuritySpy 3.4.10. No other software is installed or running on the MacPro.
- I replaced the switch and ethernet cables with new and tried various ports.
- Setting all cameras to Passive has no effect.
- Running SecuritySpy for the same cameras at the same time on both Macs has no effect on the MacPro while SecuritySpy performance on the iMac is just fine.
I have noticed that Activity Monitor reports around 190 CPU threads on the iMac, but around 400 CPU threads on the MacPro.
Any ideas?
I have been using SecuritySpy very successfully at my office building since version 1 and am now using version 3.4.10. I have added and upgraded cameras so that I now have 25 2 MP IP cameras: AvertX HD30, Samsung SNV-6013, and Samsung SNV-7084. All are H.264 formatted streams at 8 fps to Gb POE ports on a Netgear GS753TP switch through a separate private network. The switch is directly connected to my Macs.
All cameras are configured uniformly on SecuritySpy with Format: H.264 RSTP, No recompression, Text overlay disabled, Motion detection sensitivity 50, One movie per day, Frame rate: 8. The Capture destination is a 6TB RAID connected to the Mac by Thunderbolt. Both Macs are running OS 10.11.4 with SecuritySpy 3.4.10 configured identically and do not serve SecuritySpy to other Macs.
The system runs perfectly when I have everything connected to my iMac with a 4 core 4 GHz Intel Core i7 CPU and 32 GB of RAM except that I occasionally need to optimize its RAM. The iMac is also running various network monitoring and database apps at the same time.
I thought it would be best to have a new Mac dedicated to SecuritySpy which would allow further improvements so I purchased a new MacPro with 12 cores and 64 GB of RAM. All apps run normally on the MacPro except SecuritySpy. On the MacPro, SecuritySpy always starts up with all cameras showing 8 fps as expected with smooth motion in real time. After a few minutes, various cameras start showing fluctuating frame rates of 3-5, motion that lags as much as 10 seconds, and sometimes freezes. The captured video shows the major skips. Eventually the frame rate drops for all cameras, although it continues to vary, and does not differ by camera brand or model.
Troubleshooting:
- I ran hardware tests on the MacPro which showed no problems.
- I talked to Apple who said the problem is software related and confirmed that my MacPro graphics card is not defective.
- I reformated the MacPro solid state drive and installed a fresh OS 10.11.4 and a fresh SecuritySpy 3.4.10. No other software is installed or running on the MacPro.
- I replaced the switch and ethernet cables with new and tried various ports.
- Setting all cameras to Passive has no effect.
- Running SecuritySpy for the same cameras at the same time on both Macs has no effect on the MacPro while SecuritySpy performance on the iMac is just fine.
I have noticed that Activity Monitor reports around 190 CPU threads on the iMac, but around 400 CPU threads on the MacPro.
Any ideas?
Comments
-
Thanks for the excellent report and the troubleshooting you've already done, it's always helpful when we have the full picture with lots of detail.
This sounds to me like a network issue. If you look at Activity Monitor, under the CPU tab, and check how much of the Mac's resources are being used (the easiest figure to refer to the the "idle" percentage shown at the bottom), I expect that you'll find that the Mac Pro's CPU resources are not being overly taxed.
If the problem were related to the video hardware, you would see consistent sluggish performance rather than the up-and-down performance you are seeing.
Sometimes we have seen some Macs behaving better on some networks than others, and we haven't really got to the bottom of why this might be. You mentioned that you changed the ethernet cables and switch - what make/model switch are you using, and what was the previous one?
Some suggestions:
Please try tried both ethernet ports on the Mac Pro, just in case one is defective.
In the Network system preference, select the active ethernet interface and click the Advanced button. Under the Hardware tab, there are a few different options to test. Configure manually and try both the 100baseT and 1000baseT options in turn. Also, try both MTU options.
As for the thread count: to take advantage of the fact that the Mac Pro has many more cores than the iMac, many applications (including SecuritySpy) will use more threads. So the difference you see here is entirely normal. -
The MacPro is averaging about 82% idle CPU. The iMac is averaging about 41% idle CPU.
The switch had been a Netgear GS748TP and is now a Netgear GS753TP. The new switch made no difference. Using the same ethernet Cat 5e cable connected to the same Gb switch port results in good performance on the iMac and poor performance on the MacPro.
There is no difference in SecuritySpy performance on either ethernet port on the MacPro.
Now it gets interesting. Under the MacPro's Network System Preferences > Advanced > Hardware, with the default of Configure: Automatically, it showed Speed: 1000baseT, Duplex: full-duplex, MTU: Standard (1500), and the performance was poor as usual. When I applied your suggested Configure: Manually, the Speed, Duplex, and MTU remained unchanged, but SecuritySpy performance on the MacPro became perfect.
I then raised the frame rate for the cameras and SecuritySpy from 8 fps to 10 fps, and it still runs perfectly. The MacPro CPU idle has dropped to 77% which confirms its proper performance and allows future camera improvements.
Thank you!
-
VOP - are you writing that simply changing the Hardware preferences from Automatically to Manually the throughput improved and your problem was resolved?
-
I spoke too soon. After 30 minutes of perfect SecuritySpy performance on the MacPro, it went back to slow and the CPU idle went back to 84%. I have had that experience previously when it would spontaneously begin working well for varying periods and then fall back to poor performance.
Additional tests I tried with no improvement:
- Changed the MTU to Jumbo (9000).
- Connected the ethernet cable to the ethernet port on a Thunderbolt display.
- Configured the MacPro ports and the Netgear switch port connected to the MacPro for Manual Speed at 1000baseT instead of Auto.
- Placed a Netgear GS116 between the Netgear GS752TP and the MacPro.
Right now it has worked well for 30 minutes after I placed the LAN side of a Gb Cisco RV320 between the Netgear GS752TP and the MacPro. I am also thinking that it is a network issue, perhaps a problem between Gb Netgear switches and MacPros. I will update you later. -
interesting, watching this.
-
Very interesting, it's odd that it works great for a time and then starts having problems. Did you try the manual ethernet settings at 100baseT speed? This is definitely something to try. If the Mac Pro is having trouble communicating at gigabit speeds for whatever reason, it will result in very poor network performance, and switching to a lower speed may help. Judging by your setup there is no particular reason why it would be having trouble, but this would be a useful test to investigate this possibility.
-
We have spent a week repeating tests and looking for patterns. Manual ethernet settings of 100baseT on the MacPro and switch do not improve performance, neither do manual settings of 1000baseT on each. I have raised the frame rate for all cameras and SecuritySpy to 10 fps which also had no effect. When it works right, all cameras are solid at 10 fps, but usually 4-7 fps.
We focused on the network by doing data dumps from another Mac to the MacPro through the same switch. We were able to achieve data transfer rates 8-9 times the normal rate for SecuritySpy alone. We also have done many internet bandwidth tests which consistently had data transfer rates of near 100 Mbps which is higher than SecuritySpy at its best.
Mysteriously, about one in every 20 internet bandwidth tests results in SecuritySpy accelerating to its proper performance for various periods of time. Once good performance lasted 24 hours, but it would usually drop as soon as we interacted with it such as using the browser, zooming a picture or even just clicking it. Even the slow performance is inconsistent with low frame rates constantly changing for various cameras even though there is no movement present. At times the cameras will drop to 1-4 frames per second. -
hmm, is there a way to, via some esoteric configuration, remove some cores from the set SSpy can access?
you may be in thread-dump time. -
The test build did not cure the problem, but it was an improvement by making it easier to get the performance up with only a couple repetitions of disconnecting and reconnecting the network from the MacPro. Also while zooming the display of a camera no longer drops the performance, other actions such as visiting SS settings or using its browser still causes the performance to drop.
Thanks for the effort. -
I've made some further changes and sent you a new version, let me know how you get on with this one.
-
My hunch is that you're running into file, file lock or socket contention.
you could test by using launchctl to set 'large limits' on maxfiles?
or the old ulimit and launch SS from the command line?
perhaps test by turning off recording? -
I am now running today's new version of SecuritySpy. There was no improvement except that it is now easy to get SS performing properly by disconnecting and reconnecting the network which works the first time, every time. However visiting SS settings or using its browser still drops its performance until the network connection is cycled.
Making all cameras passive did not result in any improvement today. -
I have a suspicion that the problem is Netgear. I have no scientific explanation, but I have experienced performance problems on my installations that have Netgear POE switches. I have switched to TP link. No more dropped frames or broken E-pipe errors now. I will never buy another Netgear switch.
-
But a Netgear problem doesn't explain why SecuritySpy has always worked perfectly with my iMac which is connected to the same switch, and it doesn't explain why SS works fine with the MacPro until I interact with SS in a way that doesn't involve the Netgear switch such as checking SS settings.
I currently have 4 other Netgear switches ranging from Level 3 to unmanaged with a total of 160 Gb ports, and a high level Netgear router, which have worked very well. The only thing I need to keep in mind for Netgear is that managing them is better with a Firefox browser rather than Safari. -
VOP, what happens when you run
Octo:~ casey$ launchctl limit
here's mine on the computer I happen to be using now:
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 709 1064
maxfiles 256 unlimited
what is your maxfiles?
and how many files/handles does SSpy think it has open when the performance goes bad?
another question - at what point in camera count does the degradation start? have you tried, sorry, but starting at just one camera and working your way up? sounds doofy yes, but... -
I had identical results to yours when I ran launchctl limit. That is also the same result as the iMac I have been referencing that works perfectly with SecuritySpy.
My maxfiles is 256 unlimited like yours.
My MacPro shows 58 SecuritySpy files open both when the performance is good and when poor.
The degradation begins when I enable the 13th camera (out of 25), but the performance will also degrade at 12 cameras if I use the SecuritySpy browser or if most of the 12 cameras detect motion. -
sorry VOP, but that's all interesting. OK.
My investigative direction here is based on experiences I had at Stanford where we swapped in larger ( much larger ) servers and suddenly had a fallback in performance. I tracked back the performance issues to file handles and sockets in the OS.
pondering -
I was hoping that SS v 4 would help my performance problem on the Mac Pro, but the performance is no better and I lost a workaround. Under v3.4.11b4, I was able to restore full performance by disconnecting the network cable from the Mac Pro for several seconds and then reconnecting. I then have full performance until I interact with SS, even just checking the "About SecuritySpy". The disconnect/reconnect workaround has no effect for v4 so I have returned to v3.4.11b4.
-
If your router/switches have the option, I would also turn flow control OFF for the ports in the switches. Having that active on some switches with mixed 100 and 1000 mb connections can cause unpredictable throttling if flow control is on. Worth checking.
-
Thanks guykuo. Flow control was already OFF for my switch. The possibility of a socket issue from caseyd is what makes the most sense to me based on the workaround effect. Rebooting the switch also results in full performance on the Mac Pro until I interact with SS or the OS.
