"Failed to record video frame 1556,98002" error occurs every few days
  • I have an odd error I get every few days that appears to be related to the writing of the continuous capture file to the external hard disk. The error reads:

    "Error performing continuous capture for the camera "", continuous capture has been disarmed. (Failed to record video from 1556,98002 The disk is too slow and cannot keep up with wiring data - you may need to reformat or replace the disk)"

    When this error occurs, it is always the *exact same frame number*, and usually lists about 5 or 6 of the 13 cameras having this error at the same time. But this error only occurs every few days...

    I already tried doing a deep health scan of the disk which came back as OK. Disk is an external USB3-connected 2-day RAID0 setup by Western Digital.

    Thanks in advance for any help! Please let me know what I can do to help diagnose...
  • From your description, it sounds like the drive is experiencing significant transient slowdowns every few days. When this happens, data backs up in SecuritySpy's RAM buffers, but this can only be tolerated so much before SecuritySpy will stop recording and throw this error. The error represents a real problem with the drive; it is not a problem with SecuritySpy or an erroneous message.

    We see this problem mainly with RAID setups, probably because of the extra layer of complexity involved in RAID vs standard single-drive setups. In terms of the error code itself, it just identifies this specific problem, it doesn't refer to anything like a frame number.

    After this happens, SecuritySpy will attempt to re-arm the cameras and continue recording, so hopefully the loss of footage isn't too significant.

    In terms of a solution, I would suggest that instead of a RAID0 setup, it's preferable to reconfigure this as two separate drives (each in their own enclosure, each connected to the Mac via USB 3), and split the cameras between these two drives. As well as better performance, this method will give you better resilience (if one drive fails you only lose data for half the cameras, whereas in the RAID0 setup, if one drive fails, you lose all the data for all cameras).
  • Thanks Ben for the detailed response, makes sense.

    When this error does happen, it is always for the continuous recording, and it is always for many cameras at the same time.

    For continuous recording, I have it set to record / save in 24-hour increments that always end at midnight. Could it be that all 13 cameras (and their 15 GB 24-hour video files each) all attempt to get saved to the USB drive at once, resulting in this issue? I don’t see any obvious way to stagger saves to 12:01 / 12:02 / 12:03 per camera for example to try and help minimize this.

    Or does SecuritySpy already sequentially try and stagger the writes one by one? I’m curious what is actually happening when midnight hits and it attempts to save these 13 massive files at all at once, and if there is a way the software could handle that temporary condition better to resolve the problem and be more resilient to hard drive slow downs? Thanks.
  • The video and audio data is saved to disk continuously, as it comes in from the cameras. As this is happening, the metadata (frame/packet sizes and timing information) is kept in memory. When the file is finished, this metadata is then written to the end of the file as a "movie resource". So there is a bit of extra data that needs to be written at the end, but not much, and this shouldn't be the thing that is causing this problem.

    Are you seeing this error happen at midnight only? If this is the case, then it could be some other disk-related task that happens at midnight (e.g. a backup task?).

    Continuous recording is the mode that puts most demands on the disk, and all cameras that write to the same disk share the same write queue, so it's not surprising that this problem is simultaneously affecting the cameras that are capturing continuously.

    To make SecuritySpy more resilient to this type of drive slowdown, I have increased the file writing queue size in the latest beta version of SecuritySpy (currently 5.2.1b4) so please install this version and let me know how you get on with it. This uses a larger memory buffer in order to reduce the sensitivity to temporary disk issues. There is only so far that we want to push this though - if the memory buffer were too large, this could potentially cause memory problems when the disk slows down.

    So, I would still recommend reconfiguring your drive setup as I outlined above.
  • Thanks! I've installed it and will report back on how the write errors change.

    But one thing I noticed relatively quickly is this latest 5.2.1b4 app seems to lock up / crash now when I disable a camera, apply preferences, and then go back to re-enable it and a apply preferences later. As soon as I attempt to apply the re-enable preferences, I just get a spinning ball and the server goes down. This is the first build where I have seen that issue as I often disable / enable cameras. Would you mind double checking if you can reproduce that issue? Thanks.
  • I'm afraid I cannot reproduce this issue. Please do the follow:

    - Reproduce the problem
    - Open Activity Monitor
    - Select SecuritySpy in the list (it should say “not responding”)
    - Click the gear icon at the top and select “Sample Process”
    - Save the resulting log and email it to us.

    This log should tell us exactly what is happening and hopefully how to fix it.
  • Thanks Ben, just sent you a log from the non-responding Security Spy. By the way, so far after 4 days of testing I haven't seen one of the "failed to record video frame" errors. Fingers crossed that your adjustment has added enough headroom to keep it solid!
  • Thanks for sending the log, this problem should now be fixed in the latest beta version of SecuritySpy (currently 5.2.1b7). Please confirm.

    Great to hear that the new drive arrangement is working well!
  • b7 fixed the hang! You guys are good. :) thanks!
  • Great to hear that!
  • Hi Ben, just to follow up on this issue: while your changes improved the situation and reduced the error occurrence, I was still receiving the write failure occasionally. So I bit the bullet and replaced the 2-drive USB3 external storage with a newer 1-drive version (of even larger capacity). I’ve been running that newer 1-drive setup alone for 2-3 weeks now without a single error.

    So your originally theory was correct: there was some issue with the 2-drive RAID0 enclosure storage causing I/O slowdown occasionally. Fingers crossed that things stay solid as I assume the better test will be when the hard drive is at full capacity and SS is constantly swapping out oldest data for newest data.
  • Many thanks for reporting back, and great to hear that the hardware change has resolved the problem, I'm sure this will be helpful for other users experiencing the same thing. Yes, when the drive is at its limit and SecuritySpy has to start deleting files it will be under more load, but this extra load will be minor and shouldn't cause any problems for a well-functioning drive over a fast USB 3 connection.

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!