Skip to content

Error - Disks too slow reported but not the case...??

2»

Comments

  • @billie - Backblaze can write a lot of data to disk in the form of temporary compressed files that are queued for upload. You can control where it creates these files using the "Temporary data drive" setting in the Backblaze settings - make sure this is not set to the drive that SecuritySpy is using for capture.

    @cstout - smoothing applies a moving average filter to the graph for its display, which has the effect of smoothing out extreme peaks and troughs. A higher smoothing setting is more useful for seeing the overall trend of the data over time, whereas a low smoothing setting is required to see the actual values of these short-term spikes.

    As for the 650% packet loss - this should only happen if you are summing the packet loss from multiple cameras, is this the case, or are you viewing the packet loss from just one camera? If the latter then this is definitely a bug! Please could you email us a screenshot of this to demonstrate. Thanks.
  • @ben - yes, temporary data drive is set to Macintosh HD, which is a 2 TB internal SSD. Backblaze is constantly running, as I don't think it is possible to catch up due to my relatively low upload bandwidth (only 20Mbps). So I think the first option is to simply disable Backblaze from backing up my video capture volume (a Drobo 8D connected via Thunderbolt 3). I had previously disabled uploading the *.unf files, but should have disabled the entire volume once I added 4 x 4K cameras (Lorex LNB9232S). I also have the TimeMachine volume on the same Drobo. The capture rate seems to be about 100GB/day/camera, and I have 4 more cameras on back order (Lorex LNE8964AB). Ideally, the continuous and/or motion captures could go directly to an SSD, and then once closed, be moved (versus copied) to the larger capacity drive for long term storage (30 days or so). Storage would need about 24TB for 30 days, which is what I allocated to the volume. I do see "Upload to" options for each camera, and saw that local transfers were permitted (according to manual), but it is unclear if I can still use SecuritySpy to browse/view such files that have been moved (particularly if I've deleted them from the original destination, perhaps by setting the "Delete old files by age" to 1 day). Perhaps the real solution is to find some Mac OS software to automatically handle a tiered storage arrangement, with new files written to the SSD, and those files migrated automatically behind the scenes to disk, but still presenting a full view of the files to user space...
  • @ben it is also curious that everybody reports the same frame number (1556,98002 ) in their error log. is there an incorrect format specifier? I've disabled Backblaze from backing up the capture volume and we'll see how it goes. I also am running an "fs_usage | gzip > fs_usage.gz" in the background to a different device (the SSD). If I get the error again, perhaps I will be able to correlate it with specific process activity. I suspect it is backblaze or spotlight. I've also added the capture volume to the spotlight privacy tab as a pre-emptive strike...
  • @billie yeah, all of my errors report 1556,98002 in the log.

    @ben this is all I have for the last two days of log entries. As far as the packet loss graph goes, yeah, it was only for the one camera. I'll send over a screenshot.

    09/02/2019 00:12:54: Error performing continuous capture for the camera "SW Driveway", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:12:54: Error performing continuous capture for the camera "Back Yard Side", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:12:54: Error performing continuous capture for the camera "Garage", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:15:03: Error performing continuous capture for the camera "Back Yard", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:15:03: There was an error while recording audio data for camera "Front Door". 5.0.1,7460,98002 Failed to record audio chunk. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/02/2019 00:15:06: Error performing continuous capture for the camera "Front Door", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: Error performing continuous capture for the camera "SW Driveway", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: There was an error while recording audio data for camera "Garage". 5.0.1,7460,98002 Failed to record audio chunk. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: Error performing continuous capture for the camera "Garage", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: Error performing continuous capture for the camera "Back Yard Side", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:54: There was an error while recording audio data for camera "Back Yard". 5.0.1,7460,98002 Failed to record audio chunk. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:13:55: Error performing continuous capture for the camera "Front Door", continuous capture mode has been disarmed. 5.0.1,1556,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    09/03/2019 00:17:01: Error communicating with the network device "SW Driveway". 5.0.1,10,835 Excessive packet loss from network device, the network may be too slow or defective, or this computer may be overloaded. Check the network and/or reduce this camera's frame rate.
  • With both BackBlaze and Spotlight disabled on the capture volume, the problem has not recurred for me. Incoming data rate (sum) peaks at 10490 KB/s; Recording data rate (sum) peaks at 11520 KB/s; disk pressure (sum) 0; memory pressure 0; cpu usage (total) peaks at 32%; still have 4 more 4K cameras to add (still on backorder...)
  • Well it happened again. But this time, I did capture some additional data via fs_usage. I'm not sure how to interpret it though. I'll try to send some info via email.
  • Please both make sure you are using the latest beta version of SecuritySpy (currently 5.0.2b9) as this has an enlarged disk write buffer. @cstout your log entries indicate that you are still using v5.0.1.
  • :-) I looked right over that comment in your post. Just loaded up the beta. I'll let you know if I get my nightly messages.
  • For my own setup, which uses 8 x 16TB Seagate Ironwolf Pro NAS drives within a Drobo 8D (direct connect via TB3), I discovered that disk I/O to the Drobo would simply stop (independent of SecuritySpy). For example, music playing off a volume on the Drobo would simply pause for several seconds. The pause correlates to the error messages sent by SecuritySpy. Drobo support indicates that some of my drives have an unexpectedly high "busy" time, i.e. disk operations are taking longer than they should. I've moved the capture streams off the Drobo and onto a faster (but much much smaller) SSD drive while I investigate the Seagate performance issue. The Drobo itself was showing write throughput over 100MB/s at times (while running some other applications), so I know it is capable of the sub 20MB/s rate that SecuritySpy would be sending.
  • I hate to reopen this issue, but it is hitting me constantly now after upgrading to SecuritySpy 5.3.3, and Mac OS 11.3. Not sure what is going on, as I have a pretty beefed up system specifically to avoid this problem. I had started to have problems with the automatic disk cleanup, so I had exited SecuritySpy, reformatted the volume, and restarted SecuritySpy. That is when it upgraded to 5.3.3 (yesterday iirc), and Mac OS 11.3 came out today. A performance test (AJA System Test Lite) shows 1358 MB/sec write and 2191 MB/sec read which has always been fast enough. This is a 16TB SSD volume (8 x 2TB as RAID 0 APFS). The dashboard shows Recording Data Rate All Cameras (Sum) hitting a max of 19 MB/sec, far far below 1358 MB/sec ability of my volume. I did note that my BackBlaze configuration appears to have lost the Exclusion for this disk. I have reconfigured that. I also expect BackBlaze to be a bit busy after the Mac OS update, however, the system should have plenty of excess capacity for such things (yet, even now, I see the "Error performing continuous capture for camera...The disk is too slow" messages. This is running on a MacPro (2019), 16-Core, 96 GB Memory, AMD Radeon Pro Vega II 32GB.
  • Hi @billie please open the Dashboard in SecuritySpy and view the "Disk Pressure" graph with smoothing off. What do you see there? The metric, which is shown as a percentage, indicates how full SecuritySpy's disk buffer is. The buffer is generous, and when it fills up, there is really nothing that SecuritySpy can do but stop recording and generate an error. The message really does indicate some sort of problem with the drive. This problem may show up quite infrequently and may therefore not be seen with disk speed tools.
  • I think the 11.3 upgrade had a negative effect on things. I tried to erase and reformat the existing RAID set, but Disk Utility always got hung up (requiring a system restart to correct, as Force Quit'ing Disk Utility wouldn't allow Disk Utility to be restarted). I've completely removed the RAID set and re-created it, and will see if things improve. That said, the back pressure does show unreasonably high values starting April 26 at 3:30pm local (shaded to show > 95% most of the time with spikes to 100%). If I go back in time, the highest I see is a spike to 5% a few weeks ago (which is more in line with expectations). After re-creating the RAID set, the back pressure seems back to normal (zero or one percent). I do still have BackBlaze trying to "catch up" to the 11.3 install, but it is configured to exclude this RAID set, as well as my TimeMachine volume. Note that TimeMachine does include my SecuritySpy RAID set, but with 10 disks it is no slouch either. I'll monitor the back pressure and let you know if re-creating the RAID was all that it needed. It would point at some hole in the Mac OS implementation...
  • I’m seeing this after 11.3 upgrade also. DS1819+, 8 drives, gigabit. No known disk errors.
  • BenBen
    edited April 2021
    Hi @billie - interesting. Let's hope it continues to behave after recreating the RAID. The reason I asked about the pressure graph is that, in normal operation, this should always be below around 5% for a healthy disk. If you see consistently high pressure, this indicates that the disk is generally too slow. But if you see mostly very low pressure, but with infrequent 100% spikes, this indicates a disk fault (or, with network drives, a network fault), consistent with the disk simply freezing for long periods of time unexpectedly.
  • the disk pressure returned to normal last Wednesday the 28th, a full day after I had recreated the RAID. I suspect there was some really intensive housekeeping that goes on after the full system upgrade to 11.3. It could be any combination of BackBlaze, TimeMachine, SpotLight reindexing, etc. Whatever it was seemed to finish, and disk pressure has been back to normal ever since. Note that I did not notice anything unusual with cpu performance or disk read/writes from Activity Monitor perspective, so I'm at a loss here.
  • Great to hear that it's settled down. Yes, some temporary intense housekeeping activity could account for the slowdown of the disk, causing this error. Let's hope it was just that.
Sign In or Register to comment.