What External Data Storage System Do You Use?

Zeigh
edited August 2022 in SecuritySpy

What external data storage system do you use?

I just migrated from a (dedicated to SecuritySpy) 2012 Apple Mac Pro that was amazingly bombproof to a new Mac Studio. That Mac Pro was upgraded early on with a smaller SSD hard drive for the operating system, then kept packed full of SATA units for video capture storage. Now that Apple makes almost all of their computers no longer user upgradable for such storage, I usually buy the minimum system storage to save money and rely on various external storage options for better control.

For now, my Mac Studio just has a single Thunderbolt 3 external 1TB SSD hard drive capturing 22 cameras. It works fairly well. However, I need more stored video in the queue before it is overwritten and thinking of stepping down to a USB-C 3.2 (5 Gb/s) to SATA III multi-bay RAID enclosure. Slower, yes, but still overkill for the purpose and more cost effective overall.

Anyhow, feel free to share you own experiences.


Peace, Dr. Z.

Comments

  • I started out with a Synology NAS for Plex. Then I wanted cameras which Synology can do, but decided to move everything, including any upcoming cameras, from the NAS to an M1 Mac mini. I briefly looked into adding RAID to the Mac via software, as opposed to dedicated RAID hardware. Actually, I looked into all the storage options. In the end, I decided to simply hang multiple, cheap, external HDD's and 1 Thunderbolt SSD off the Mac. For SS, I record everything to the SSD. I then use Carbon Copy Cloner to backup everything, which includes copying over my SS footage to one of the external drives where I can store as much as I want. The SSD stores 10 days, but every day CCC copies it over to one of the HDD's

  • Personally, I go for individual external HDDs connected by USB - multiple of these for larger systems, with the cameras split across them. It's an easy and inexpensive solution, and provides a certain amount of redundancy - if one drive fails it doesn't bring down the whole storage solution (unlike RAID-0), don't affect all cameras, and is easy to rectify by simply swapping out that one drive. Any USB 3.x protocol is easily fast enough to cope with a couple dozen cameras capturing continuously to each drive, or many more when capturing based on motion. Desktop Macs give you 4x USB-A ports, and it's easy to add more via Thunderbolt/USB-C hubs if necessary.

    Many customers use RAID setups and they can work well, but they add complexity and therefore failure modes to the system. When customers ask us for help with drive issues that they have had trouble solving themselves, it often turns out they are using RAID setups, and it's then often difficult to pinpoint and fix the problem. One advantage of some specific RAID setups (e.g. RAID-5/6) is that they offer whole-storage redundancy, so that if a drive fails, you don't lose any data. However when this happens, and for a significant period after the failed drive has been replaced while the RAID is recovering, it tends to slow down the whole unit significantly, which can lead to problems capturing video during this period.

    I would say that RAID units are typically the best solution when you really need huge amounts of storage, or whole-storage redundancy, otherwise individual drives are the way to go.

  • Zeigh
    edited August 2022

    As always, thanks for your expert advice, Ben!

    Yes, I did experiment with all the different RAID configurations a few years ago (using Apple's DiskUtility or Old World Computing's SoftRAID). They all have their pros/cons. Then I gravitated to simply backing up all data to a Mac server once a day. HA, sure enough, I lost some video of an important event when that main hard drive spontaneously failed (and before the daily data transfer). So, I started using a RAID 1 (mirroring) configuration with my older Mac Pro from then on. Now that I have migrated everything over to my new Mac Studio, I plan to implement the same storage configuration via a multi-bay external hard drive enclosure.

    That brings up another line of questions:

    1. Are hard drives specifically designed for video surveillance worth the extra cost? I did start using a Seagate SkyHawk SATA hard drive a couple of years ago and it failed within 16 months. Thankfully it was covered under warranty and I want to think that this was just a fluke. The standard RAID 1 hard drive that I purchased at the same time and had matched to it is still performing just fine.
    2. Would it be worth it to have both volumes the same higher design standard? Regardless, maybe an extra measure of safety would be to stagger the replacement of each unit in a mirroring RAID.
    3. Realistically, when should such video surveillance specific hard drives be replaced? I have read anything from yearly with high security settings to every 3-5 years for residential uses.


    Peace, Dr. Z.

  • Ben
    Ben
    edited August 2022

    Thanks for reporting your experiences! Yes, HDDs can be relied upon to fail every now and then, and of course especially when there is an important event to record :) RAID 1 is a good solution to provide redundancy in case of a single-drive failure. To answer your specific questions:

    1. I have read the claims by the manufacturers, but have seen no real-world test data comparing surveillance drives with standard drives, so at this point I'm basically agnostic on this. It's plausible that designing drives with certain characteristics will make them better for video surveillance, though the lack of test data means that we basically have to rely on manufacturers' marketing claims.
    2. Do you mean multiple volumes in a RAID? I've always gone for the advice to use identical drives, though this of course is not strictly necessary. High-spec drives aren't too much more expensive than low-spec ones, so it's worth spending the bit extra.
    3. Personally, I wouldn't replace drives too frequently, if they are working well. Once you have a drive that has been working well for 3 years, it's quite likely that it will continue to to work for a few more. Replacing it just means that you're replacing a known good drive for an unknown one. Perhaps after 5 years or so this makes sense, as the probability of failure then starts to increase significantly.
  • I use a 16GB RAM M1 Mac-mini with a 40Gb thunderbolt hub. I have 3 NVMe (2 1TB and 1 2TB) external 10Gb drives plugged into it and a 10Gb ethernet adapter. I have a Synology NAS 1621+ with 32GB ECC RAM and six 10TB HDD and two NVMe drives mirrored for caches and BTRFS metadata pinning. I use file integrity settings and scheduled scrubbing because I dislike bit rot issues. I have two 14TB external drives for backup. I do not back up my video security files. For sequential transfers I can barely exceed 8.5 Gb which is pretty good or just over 1GB/s. For my NAS I shuck Western Digital drives and use the HUGO utility to convert the drives to 4Kn since that's what they are instead of leaving them in the 512e emulation mode.

    I send the videos to the NVME and use the upload feature to send the videos to the Synology by using the local feature.

    I recently read another thread where Ben mentioned you can send the motion database to the NVMe and the video to a NAS by creating two camera objects and it uses one license. So I plan on reworking this since I had no clue I could do this.

    For some reason occasionally the Mac Mini loses connection to the NAS and I just reboot the MAC to quickly fix it and it reconnects at logon again. I would rather have SecuritySpy fix that connection for me or use some other method without having to do crazy stuff in the shell. I do have to say SecuritySpy is great at sending me emails when things screw up (which is not often).

  • Zeigh
    edited August 2022

    T1533 quote: "I recently read another thread where Ben mentioned you can send the motion database to the NVMe and the video to a NAS by creating two camera objects and it uses one license. So I plan on reworking this since I had no clue I could do this".


    Wow, you have sparked my interest with this detail. Unfortunately I can't find the post you are referring to or other information. Would you (or anyone) mind sending the link?


    Peace, Dr. Z.

  • I think it's this thread that @T1533 is referring to: https://bensoftware.com/forum/discussion/3983/tiered-storage-browser-scanning-multiple-volumes

    In terms of the NVR coming back online after it disconnects - SecuritySpy does constantly attempt to do this, so I suspect that when this happens there is some deeper problem with this that is resolved by the reboot of the Mac.

  • The cloud storage company Backblaze recently published a blog post called Hard Drive Life Expectancy, which has some useful information related to the above discussion. Key takeaways are:

    • Failure rates between different brands/models of HDD drives vary considerably.
    • For some drives, failure rates increase significantly after year 4; for others, failure rates are fairly linear all the way up to year 6.
    • While it used to be the case that first-year failure rates were higher than year 2/3 failure rates, this does not seem to be the case with modern drives (perhaps due to better manufacturer quality control to weed out any drives with obvious defects that would cause early failure).
    • It should be noted that these results don't come from a random sampling of drives; Backblaze chooses their drives very carefully, so it's a bit difficult to know how these results generalise.

    So I think the takeaway from this is to:

    • Try to choose known high-quality drives with good ratings/reliability data to start with (don't skimp) - see if you can find failure data published by Backblaze or others
    • For critical applications, replace the drives every 4 years even if they show no signs of failure
    • For less critical applications, replace the drives every 6 years or so

    (BTW Backblaze is a great cloud/backup service - we use their automatic backup solution ourselves!)

  • Since I have the NVMe write cache now I may just send everything to my NAS and see how that works. My primary Windows workstation never disconnects so I don't understand what is up. My ISP supports IPv6 Prefix delegation and I can see my Windows system is using an IPv6 connection so I may turn that off on the NAS and see if it helps. The IPv4 is static but the IPv6 is DHCP.

    I don't have my NAS credentials in SecuritySpy anywhere, is that possibe? I have a share setup just for video security with its own credentials. Zeigh, sorry for somewhat derailing your thread.

  • The NAS credentials are stored by the system/Finder when you first mount the NAS on your Desktop (provided you tick this box). They aren't stored by SecuritySpy - when you set the NAS as a capture destination in SecuritySpy, it saves a bookmark to that location on the file system, and subsequently when SecuritySpy attempts to access that location, the system springs into action and mounts the NAS with the saved authentication that it has stored.

  • I use a Synology RS1219+. Works fine (10GB ethernet). SS is running on a M1 Ultra Mac Studio, 50 cameras.

    The only issue I have is every day at midnight, I get:

    Generated by the SecuritySpy system named "SecuritySpy Web Server" at MySSServer

    Error performing motion capture for camera "Aisle E East", motion capture mode will now be disarmed. 5.5.0,1554,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk. 

    Error performing motion capture for camera "Rear", motion capture mode will now be disarmed. 5.5.0,1554,55340,98002 Failed to record video frame. The disk is too slow and cannot keep up with writing data - you may need to reformat or replace the disk.

    etc. for every camera.

    Presume it is to do with the closing of the Continuous Capture file for all the cameras.

    Are others having this problem with NAS drives?

    Have considered using the Mac Studio's internal storage, but was concerned about excessive writes to the SSD. It's not like it is replaceable! Thoughts?

    James.

  • Hi @jlbrown this is unusual and shouldn't be due to closing the files (though I suppose its possible). Do you have any other task related to the disk that is scheduled for midnight (e.g. a backup or any other activity that reads/writes from this disk)?

  • Mech33
    edited August 2022

    Yes, my setup does that about once every day or two as well (recording up to a Synology NAS). I just ignore it.

  • Hi Mech33. My Synology was fairly full. I followed these instructions from Bensoft Support:


    - Quit (or force-quit) SecuritySpy.

    - Unmount all NAS instances on your Desktop (drag them to the Trash).

    - Disconnect both Ethernet cables from your Mac, so that it is not connected to any network.

    - Open SecuritySpy; it should now open without hanging.

    - Select the menu option at SecuritySpy > Debug > Clear All Camera-Specific Capture Destinations.

    - Select the menu option at SecuritySpy > Debug > Clear Old Backup Files.

    - Go to Preferences > Storage, and click the "Reset" button.

    - Reconnect both Ethernet cables, and mount the NAS in the Finder.

    - In SecuritySpy, under Preferences > Store, click the "Set" button and select the NAS.

    - Quit and reopen SecuritySpy.


    Only difference was that I told it to use the internal SSD rather than the NAS going forward. All works fine now.


    But I will reconsider recording to the Synology NAS in the future - just glad got everything working. Thanks again to Ben for the excellent software and support.


    James.