Poor performance after macOS 10.11 -> 10.12 upgrade
  • I've been running SecuritySpy v3 & v4 on a Mac OS 10.11 VM for a really long time. This has worked great. I noticed dramatic improvements in performance and system responsiveness after upgrading to SecuritySpy v4. This past weekend I installed a 10.12 VM along side my 10.11 VM and migrated all of my things, including SecuritySpy.

    This new VM runs fine for the first 1-2 hours but then VTDecoderXPService slowly consumes away all the CPU power in the host. If I restart SecuritySpy the CPU usage on the host drops to about 60% utilized. The VTDecoder process shows about 300-350% CPU used in Activity Monitor; the VM shows about 50-60% idle CPU. This is normal, and good, and works well.

    Give it an hour or so and all the numbers above double. VTDecoderXPService consumes 700% or more of the CPUs (only 8 cores), the VM shows < 15% idle CPU and the host becomes nearly unresponsive with a maxed out CPU (90%+ utilized).

    Any idea what is causing this? How do I narrow it down and troubleshoot further? I'd like to stay on 10.12, but this behavior is certainly making me want to revert. The other VM can be booted right back up and brought into service pretty easily. The other option is to create a timer/cron to restart SecuritySpy every hour or two; uhg.

    The only other difference is the SecuritySpy version. Offhand I'm not sure what version the 10.11 system has installed but it's not 4.1.1. Probably 4.0.8 (the releases don't contain dates on the website):

    Thanks!
    -franky
  • Those seem like very high numbers for decoding service. How many cameras? are they all the same make/model? Are they wireless or wired?
    is anything like text overlay turned on? (from the manual "Generally, if you require a text overlay, transformation or blanking mask, it should be applied in the camera itself (if available there), rather than SecuritySpy, to avoid the need for SecuritySpy to recompress the video data.")

    I run 10.12.x (currently .3) although directly on a quad i7 mac mini. With the particular cameras I have (all but one wired), I've never had the increase in system resources happen.
    There are other posts in the forums of it happening to some people, not sure if a cause was found for all of them (I think for at least one person it was wifi cameras with weak connections causing decoding issues from bad data)
  • Hello Brian,

    Thanks for your reply and spending some time helping me diagnose the issue.

    Some background: I'm a systems engineer regularly writing code in many languages running on linux servers; Ubuntu and CentOS currently. I'm very good at this stuff and streaming video is my passion. I happen to work for the largest live video streaming service in the world. Very awesome stuff!

    You asked about my specs, so I'm going to put it all out there. I've never actually published my specs. ;) This will reveal the secret sauce for how to run a massive system with a low budget. I've had as many as 8 3-4MP cameras connected at one time running @ about 50% CPU load. This thing is powerful and should scale to 12+ cameras just fine.

    My hardware setup:
    - SuperMicro X10SLM-F motherboard w/ 32GB ram & Xeon E3-1241v3 CPU.
    - VM storage is on a Samsung Pro SSD.
    - Camera storage is on a 2.5" 1TB laptop hard drive. I have a 750GB vmdk on this drive, mounted at /Volumes/Storage
    - Side note: SecSpy never writes at over a couple dozen MB/s (maybe 20-30 max). The drive can manage over 60MB/s and has never suffered performance issues.
    Build cost under $1000 w/ CPU mark > 10,000.

    Software Setup:
    - ESXi 6.0 + vSphere for management. (~$1000)
    - Two other VMs: pfSense (low usage doing network router) and Mythbuntu (network TV tuner + samba repository of media, also low usage).
    - Brand new 10.12 VM (all updates installed).
    - Other software running on the mac VM: Indigo 7. Indigo does a lot and I can in no way call it low usage. It's pumping data into a local postgresql installation, as well as graphite w/ a grafana front-end. Indigo is the interface into my kingdom so to speak. In addition to that short list of apps, I'm running Ubiquiti Unifi (WAP management software), Skype and Messages.app. I also installed homebrew and Xcode Cli tools. The only thing homebrew is actively running is Grafana.

    Always Running:
    - IndigoServer
    - SecuritySpy
    - Grafana (`brew install grafana`)
    - Graphite (https://github.com/davidnewhall/indigraphs)
    - PostgresAPP (https://postgresapp.com)
    - Ubiquiti Unifi
    - Skype
    - Messages
    - Activity Monitor (only lately....)

    I will say that Indigo uses a bit of CPU and so does graphite, but this same software runs on the 10.11 VM as well. I don't have any specific problems with these additional puzzle pieces. Restarting SecuritySpy on a regular interval simply resolves my problem and brings the load way down on the server for an hour or so. I have screenshots of all 8 cores completely maxed out. I also have screen shots of the weird dips the precede the jumps in cpu usage. I'm not sure what to make of it. I really don't know how to dig deeper into this video decoder.

    Security Spy Setup:
    I do no transformations or re-encoding in SecuritySpy. I currently have 6 cameras in the mix. Five are wired Dahua and TrendNET cams between 3 and 4MP. The sixth is an unsupported piece of crap pushing out 1080p(2mp) using onvif over ethernet. The cameras and the mac VM are on a dedicated VLAN w/ no wireless clients.

    All 6 cameras have motion detection turned on, and they run an applescript that sends me pictures when there is motion - https://github.com/davidnewhall/iMessageSpy - This all works normally. At least it's normal until the load goes up, then all the pictures are green - like the video. SecSpy also records video when there is motion. No cameras do any time lapse or continuous recording. I have pretty basic settings to keep operations in SecuritySpy as lean and fast as possible.

    I hope some of this helps, but honestly, I've got my 10.11 VM back up and 10.12 shut down already. I wake up every morning to a non-operational TV and really slow Internet; I just can't do it anymore. I suppose it's my fault I run all these things on one computer, but I promised myself I'd keep it all under 300 watts. Thanks again!
  • All wired cameras should rule out "weak"/intermittent wifi that has caused issues for some people. And you have the options set so it shouldn't have to do much work.
    If you can be without a camera at a time, it would be time consuming - but useful troubleshooting to disable one camera a day to see if one of them is causing the VTDecoder to run amok. (VTDecoderXPCService is an OS function doing the grunt work for Security Spy 4.x - if it's getting data it doesn't like, it can cause the extremely high CPU load and/or impact the rest of the computer by consuming resources, doing a google search for the service you can see examples going back to Lion of it causing problems with run away CPU usage for various reasons)

    My 6 camera's are lower resolution (mostly 720p H.264) - at night Security Spy uses about 9-10% on average, during the day a bit higher, no more than 20-25% usually. with similar day/night CPU percentages by VTDecoderXPCService. 2 Camera's doing continuous recording, rest only on motion (3 of the cameras on schedule so they don't trigger or record when we are in the house)

    (power is one of the reasons I like this 2012 Mac mini - idles at about 10 watts, typically 13-15 watts - the USB 4 bay enclosure with 4 drives uses more power, as do the screens when they are on, which is only a couple of times a day for maybe 30-60 minutes, and the FreeNAS in the basement that uses about 200 watts - similar specs to your main system although has 14 drives in it)

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!