V5 Slow Web Server
  • Hello,

    I looked in the discussions and saw no other reports of this, but I assume most people don't hit the web interface as much as I do.

    This request can take up to 5 seconds sometimes.
    time curl "http://127.0.0.1:8000/++image?cameraNum=1&width=375&height=281&quality=1" > /dev/null

    The quality, height and width options don't seem to change the response time much. Sometimes it's very fast at around half a second, but usually it's over a second.

    CPU running around 40% - most of that decoding cameras to do motion detection.

    If I close SS5 and open SS4 and repeat the request a few dozen times I might get 1 or 2 that take over a second. Is this fixable?

    Thanks!
    -captain

    PS. This just gave me the idea to stick response times into a time series database and graph them.
  • Provided the CPU usage of the Mac is reasonable (40% is fine) then the response time on this should normally be very fast.

    Two possible causes:

    1. If the incoming frame rate of the camera itself is low, then this will increase the time that SecuritySpy has to wait for the next frame before it can be delivered via the web server. What is the frame rate at which the camera is sending video to SecuritySpy?

    2. If you have checked the "Decompress incoming video frames only when required" option in the General preferences, this can significantly increase response times, as then SecuritySpy has to wait for the next I-frame in the stream before it can satisfy the web request, which may take several seconds. Do you have this option checked?
  • 1. This appears to happen on all of the cameras. They all run at least 20 fps. Camera 1 is actually currently configured to run 40fps with a 1s (40 frame) i-frame interval.

    2. It was checked. Unchecking that box seems to have made it worse. Definitely didn't get any better.

    I'm going to play around with fewer cameras enabled. My GPU (MSI GeForce GT 710 2GB) runs about 50% GPU usage and 75% memory usage. The CPU is 50-60% idle on average. I have 9 cameras enabled - one of them is a Logitech webcam, the other 8 are Dahua IP cameras 4-6MP.

    Additional detail: when I receive an image using the web interface as shown before (except redirect to a file instead of null) any motion in the image is almost always distorted. The same distortions are not present in the captured/saved video files. Examples (I have thousands more....):
    https://share.getcloudapp.com/mXum9KEp
    https://share.getcloudapp.com/5zuJgNDd
    https://share.getcloudapp.com/L1uknrzJ
    https://share.getcloudapp.com/KouexJLB

    This same distortion is now happening in SS4 too, so I believe the macOS upgrade is part of the culprit here. Thanks for the help!
  • Welp. I hate to say I found the problem, because this has been bugging me for so long, but I think it's one of the cameras. SecuritySpy doesn't indicate any particular issue, but disabling my Pool camera has sure improved the system performance overall.

    It's a IPC-HFW4631F-ZSA which turns out to be a firmware hacked (and locked) chinese-version camera with firmware from 2017. It supports H264H and an incompatible H265. I suppose I'll be replacing it shortly.
  • thanks for the update. did you disable each camera in SS, or simply unplug each one at a time? or?
  • I disabled all but 2. Tested. Things were good, then added them back one at a time. The Pool camera made the biggest difference. It's an oddball in my system. The rest are newer models with firmware from 2019.

    EDIT:
    I'm still getting video distortions. :(

    ...and, I just unchecked "Enable this camera" in the camera settings.
  • The video distortions look like decoding issues, and could be related to the slow web performance you are seeing. This seems to happen on newer macOS systems when the GPU gets overwhelmed with the decoding task. If you open Activity Monitor and select "GPU History" from the Window menu, what kind of GPU usage are you seeing?

    Try this: click on the SecuritySpy menu, and from the Debug submenu, check the option "Disable Hardware Accelerated Video Decoding". The quit and reopen SecuritySpy. How does this affect CPU usage, GPU usage, web response time, and the distortion issue?
  • The GPU runs about 75% average. sometimes spiking above that. I have another diagnostic app that makes it more clear than activity monitor. Activity monitor helped me prove it's giving me the real numbers. Quitting SS makes the usage drop to 0%. Checking the "disable hardware acceleration for decoding" in the debug menu (then quitting/reopening) did not change the GPU usage. When I quit SS it dropped to zero, but it went right back to 75% after re-opening SS. This is without camera windows being shown on the screen. Version 5.2. I tried checking both boxes (encode and decode), but the GPU always goes to 75% when SS opens. This sure feels like part of the problem.. Any ideas?
  • Here's a picture of the system stats. Maybe it's useful.
    https://share.getcloudapp.com/X6uDWGeJ
  • Would a better video card help? Any recommendations? Quadro P2000 or P4000 maybe?
  • Something different happened earlier today. The CPU and GPU usage dropped significantly, but SecuritySpy is still doing its thing. The curl request in the original post is taking 5-6 seconds regularly. I made a video to help demonstrate what the system looks like. This shows the video distortions; pay attention to the baby cam in the bottom left. The road cam in the top right doesn't ever seem to distort, but most of the others do. The camera in the top right only runs at low fps, or it begins turning green. I just have a plethora of problems. Any help is appreciated!

    The video:
  • I'm somewhat perplexed by these symptoms - the video decoding artefacts you are seeing are normally associated with a system that is overloaded, but your CPU and GPU usage are well within normal ranges.

    Would you be willing to allow us access to your Mac (e.g. via TeamViewer) so that I can connect and run some tests? I think this is going to be the best way forward. If this is OK, please email support@bensoftware.com with the TeamViewer details and let me know when would be a good time to connect.
  • Sorry for the slow reply, but I've literally spent a week coming up with a fix for this. I don't know exactly what caused it, but I can surmise that it comes down to macOS is not a server OS. Using it like one leads to problems like this. In addition to security spy, this server is running: Indigo, InfluxDB, Grafana, nginx, unifi controller, unifi-poller, motifini, messages, little snitch, time machine server, backblaze, glances, telegraf.

    I found an old POS system in my closet, slapped a couple drives in it and booted up unRAID. I then moved unifi controller, unifi-poller, nginx, grafana, telegraf, influxdb and the time machine server out of my security spy system and into the new unraid system as docker images.

    The security spy system now runs: indigo, motifini, messages, little snitch, glances and backblaze. I'm soon going to remove the backblaze app too, and setup a daily cron job to move captured security spy files to the backup server on my network.

    I'm still in a weird situation with the limited authentication abilities within the securityspy's web server. I have to run little snitch to create an inbound firewall to limit who/what can access the unauthenticated security spy APIs. Indigo is in the same boat, with very archaic and limited web auth configurations. Once these apps provide better authentication and authorization inputs I can remove little snitch.

    Motifini will always be on my system, but I will probably write a pushover (or telegram) integration to remove and replace iMessages.app as my notifications platform.

    I use glances to put system metrics (disk, cpu, ram, CS, intr, net) into influxdb (on a remote host). I'll probably put telegraf back because glances doesn't give me everything I need/want.

    In the end, I believe InfluxDB and the unifi controller were the culprits. Grafana's image renderer plugin probably wasn't helping either. All of these things create many quick CPU spikes and lots of context switching and interrupts. While my CPU wasn't maxed out, I think the OS was.

    I'm now running with 8 IP cameras enabled, motion detection enabled on 5 of them at about 60% idle CPU. The GPU runs at 40-30% idle.

    I have no more distortions and the API always returns in under 1 second.

    Thanks!
  • Great to hear you found a solution to the problem. Yes, it makes sense that the other processes running on the machine were causing unpredictable CPU spikes, leading to these problems.

    In terms of the authentication issues you mention, could you please outline the problem and maybe I can suggest a better solution than the use of Little Snitch?
  • Even after clearing many other apps from the system I continued to have distortions and a slow web server. It took me a while of digging and probing, but I found many log lines like this in SecuritySpy:

    01/01/2020 01:26:36: Error running neural network. 5.2,2060,15802 Vehicle classifier failed. CoreML prediction error.
    01/01/2020 01:56:16: Error running neural network. 5.2,2060,15802 Vehicle classifier failed. CoreML prediction error.

    Disabling human and vehicle detection has completely solved my problem and returned my system to normal behavior. Hope this tip helps someone else! Good luck!
  • We've never seen this error before, though it does make sense that this might be causing problems with other features of the software, as the AI stuff can make heavy use of the CPU. How often were you seeing these message? What macOS are you running? It may be worth updating to the latest macOS to see if this resolves the problem.

    If you go to the Debug submenu via the SecuritySpy menu, and select the "Run Classifier Test", do you see correct predictions, or errors reported there?
  • Running macOS 10.13.6. Those errors seemed to be all over the log file when I looked a week ago, but they're not there now. Stupid question, but where are the old log files so I can go look closer? I just happened to notice the messages in the web browser interface. Upgrading this system isn't really an option right now, but I did buy an iMac to replace it before I "fixed" this problem. I'm going to get that switched over in the next week or two as I have time; and it'll be on Catalina (unless you suggest Mojave - the primary function is stable security footage).

    As for the classifier test: looks like it throws errors. See screenshot: https://share.getcloudapp.com/5zuJNKmm
  • TwitchCaptain, have you increased the ulimit on your mac?
  • Yeah, had to long ago when I put influxdb on it.
  • gotcha.
    I am running Jira, Confluence, SSpy and a mess of node on the same box, and had to jump it up. ( also many file handles for NAS storage and DB support )
  • For sure, I used to run all my things on the mac server too, but it got burdensome. I now run a dedicated unraid server with all the things connected there. Works out better. The mac only does sspy and indigo now.
  • Just installed Catalina on a 2015 4k iMac 21.5" i7 3.3Ghz 8GB ram.

    Then installed SecuritySpy, added 2 cameras, and immediately saw the same video distortions on my Dahua speed dome. It's running at 40fps 1080p. I'm convinced there is nothing I can do to fix this, and the problem lies in the interactions between SecuritySpy and the newer MacOS. This system is literally doing nothing except running SS, and the video preview window watching a car drive by looks like soup.
  • Does the "Disable Hardware Accelerated Video Decoding" option have any effect on the video distortions that you are seeing? Remember you have to quit and reopen SecuritySpy for this to take effect.
  • 40fps 1080p <--- that is a pretty high frame rate for a surveillance system. The fastest I let any of cams run is 15 and that is for LRP duty.
  • I'll give that a shot Ben, but I've also determined this iMac has a problem. I'm about to buy a brand new 2019 model from Apple directly, and will let y'all know how it goes.

    As for 40fps, that's not even enough. I want to run it at 60 fps, but SS pretty much chokes up on anything north of 30. I've since set the camera to 30fps/4MP. It's better, but I'm still regularly seeing distortions on jpeg snapshots and in the camera display window(s) directly on the iMac. This is affecting several cameras, but doesn't ever seem like it gets them all at the same time. Perhaps 1 or 2 of them use the video card at a time, and it's part of the problem? We may find out..

    This iMac runs at roughly 70% idle CPU and the video card hovers around 75% idle.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!