Skip to content

External Drive for storage

DuHadsDuHads
edited November 2020 in SecuritySpy
I'm considering the purchase of the new M1 Mini as my solution for my new home security camera system. I want to get an external drive to store the video. Does anyone have any recommendations for this? Is a USB3 drive sufficient, or should I get a USB-C/Thunderbolt drive?

Comments

  • WellsykeWellsyke
    It may depend on the number of cameras you have and their video formats but I've been using a 2.5" USB3 external drive with a 2014 model mac mini and 4 cameras for over a year without any issues. I picked a bus powered drive to use less power and share the Mac's UPS.
  • calvarezcalvarez
    I have SS recording to an external USB 3.1 SSD which works just fine for 8 cameras. They vary from 1920x1080 resolution up to 3840x1920. They are all set to medium-high quality.
  • BenBen
    I would say USB 3.0 should be your minimum. This provides plenty of speed for even the fastest external drives. So you could either get a USB 3.0 drive and connect it to one of the USB-A ports on the mini. Or you could get a USB 3.1 drive and connect it to one of the USB-C ports on the mini. The price difference between these two options shouldn't be too great, so it just depends on which port you want to use for the drive and which ports you want to leave available for any other devices you might want to connect.

    Thunderbolt drives/enclosures are more expensive than USB enclosures, and there's no point in the extra speed (40 times faster than a fast SSD, and hundreds of times faster than the fastest HDD).

    The other choice is whether to go for the SSD (fast but expensive and reliable) or HDD (slower but cheap). If you're mainly doing motion detection, rather than recording continuously at high frame rates, then a standard HDD drive would do the job. In this case, I would recommend a 3.5" drive over a 2.5" drive, as these tend to be faster, higher capacity, and more reliable in the long run.

    Note that the terminology is confusing, and drives can be mislabelled. USB 3.0, USB 3.1 and Thunderbolt are all protocols, whereas USB-A, USB-B, USB-C are physical port specifications. USB 3.0 tends to use USB-A ports, whereas USB 3.1 and Thunderbolt tend to use USB-C ports. Sometimes, USB 3.1 drives that use USB-C are mislabelled as Thunderbolt.
  • ComaComa
    Buffalo DriveStation Axis Velocity High Speed External Hard Drive 3 TB with USB 3.

    I bought this on Amazon for around $90. It seems to work fine.
  • BenBen
    edited November 2020
    This ↑ looks good for general purpose usage. Reliable brand, high capacity, 7200 RPM, 3.5-inch, USB 3.0 and inexpensive.

    Also: before usage with SecuritySpy, I would recommend using Disk Utility to format the external drive with "Mac OS Extended (Journaled)" format.
  • ConvenIntegConvenInteg
    edited November 2020
    A client with twelve cameras is asking to move from motion detection only to 24/7 recording. I think he would need at least five days' retention. Currently using low resolution cameras and moderate frame rates but as time moves forward, will be replacing with 4K in mind.

    The USB terminology is a bit confusing. I could imagine a four bay RAID 5 array, perhaps based on 7200 RPM "surveillance grade" hard drives. SSD drives, not just prohibitively expensive, might not be desirable in a continuous read-write setting, not sure.

    Any recommendations for very large capacity direct attached storage for Mac Mini M1 for 24/7 recording? In this case is USB 3.1 recommended?

    Given a properly structured gigabit network, would a NAS (=network attached storage) perform SecuritySpy interaction services just as well?

    Thanks all for any feedback you may provide.

    PS: On edit, I wanted to add my own experience based on older hardware, and I guess I'm wondering what may have changed. In my installation I have twenty cameras, 1080p and 4K and a (now very old) Promise Pegasus R6 (RAID5 10TB usable) connected by Thunderbolt 2 to a 2015 iMac. SecuritySpy files had been recorded flawlessly to this direct attached storage for years. More recently I've chosen to run SecuritySpy on a different iMac and so SS files are being recorded to the RAID by file sharing over the network, and I've noticed no change in performance.
  • BenBen
    Hi @ConvenInteg - I think with continuous recording it's better to go for HDDs rather than SSDs simple because you'll need a lot of storage capacity. According to our System Requirements Calculator, with 12x 4K cameras capturing H.265 video continuously for 1 week, you'll need 5 TB. This can be achieved with a single drive. However 1 week isn't very long, you may want to discuss with the customer about increasing this requirement.

    You might want RAID for redundancy, to minimise the risk of data loss in the case of a drive failure. RAID 5 would work, but this requires a minimum of 3 drives, and write speeds aren't great. For the system you are describing, RAID 1 (mirroring) with two drives is probably best - fast write speeds and 1-drive redundancy.

    Customers do use NAS devices, and they can work well, but I wouldn't recommend them if there are reasonable options for direct (USB, Thunderbolt) attachment. The network connection is inherently less reliable than a direct bus.
  • ConvenIntegConvenInteg
    Hi @Ben, thank you for the detailed guidance, much appreciated. I also want to acknowledge it had been awhile since I visited your System Requirements Calculator and -wow!- all the answers are there. What an incredibly useful resource, thank you very much!
  • DuHadsDuHads
    @Ben, I thought I read somewhere that continuous writing to an SSD was not a good idea. I could be wrong, and maybe I mistook an SD card for SSD drive. But, continuous recording to an SSD is ok?
  • BenBen
    Hi @DuHads, as far as I understand it, the problem here is the finite number of write/erase cycles that SSDs can do. This limit will eventually be reached if you are (over)writing large amounts of data constantly. However, with a drive of the correct size for SecuritySpy, you'll only be overwriting one whole drive's worth of data per month or so, therefore even a standard SSD should last several decades.
  • NeurobitNeurobit
    ^^^ This ^^^

  • ComaComa
    I should add that I’ve used a Macsales 2TB disk in Raid 1 for Time machine backup for several years. 3 year warranty that was test with a drive fail. It was replaced with a then current unit. They offer a drive free case, too. No affiliation with them.
  • donbvonbdonbvonb
    I bought the 2TB Mac Mini M1 in the hopes of using the main drive as cache drive for the video, would it make sense to boot from an external USB 3.0 drive to give your software priority over the solid state for this purpose? I would like to record 43+ cameras and have the ability to archive video automatically, if possible, otherwise I would just use some form of automation. If Files disappear, does that mess your program up or does it recover from external influences such as a monthly move to external device. Would the videos remain indexed (I looked briefly and noticed there is the capability to designate data destination PER CAMERA but what is the technique to maximize retention with expandable storage options (network shares?)

    Thanks and I have a lot of reading up on this to do, thank you for providing this discussion forum to learn from.
  • BenBen
    Hi @donbvonb I would recommend booting from the internal drive in all cases. As for SecuritySpy's storage, you have a few options here:

    1. Add a large external (USB 3.0 or 3.1) drive, and set SecuritySpy to capture directly to this drive for all cameras. This can be set under Preferences > Storage. This could be a single large HDD or a RAID setup (RAID offers potentially a lot more storage space, as well as fault tolerance). You can use our calculator to determine a good size for this drive.

    2. SecuritySpy records to the internal drive and it uploads files automatically to an FTP or Amazon S3 server. This is a good way to provide a backup of recordings with a long history in case something happens to the recording machine. The FTP server could be a NAS on your local network, or an Internet server (though for so many cameras, an internet connection with a very fast upload speed would be required for this). Note that SecuritySpy can't manage the storage of files on FTP servers (i.e. can't auto-delete old files) so this would have to be done by some other means.

    3. SecuritySpy records to the internal drive, and you implement some kind of archiving method, with some script/software that automatically removes old files from the Mac's internal drive to some archival drive. This could work well, though SecuritySpy wouldn't be able to access files on this archival drive; the files would just be there for manual access just in case you need them. So SecuritySpy would essentially see a history up to the 2 TB size of the internal drive (which should be a reasonably long history if you are using Motion Capture and not Continuous Capture).

    I would say that option 1 is usually the best: you get a large amount of storage, which SecuritySpy can manage, and which SecuritySpy has complete access to. And you don't need to use any add-on scripts or apps to manage/archive files.
  • donbvonbdonbvonb
    edited December 2020
    so I'm targeting option #3. I have a G-Tech 28TB External that with 43 file writes gets around 350MB/s but with 1 gets 480MB/s using your tools (iirc) so I think I'm going to just set up an Automater task to rsync your Captured Files folder over to external drives, and then replace themselves with symlinks? Does that have any chance of working or am I wasting time heading this route?

    I think I would be able to get higher FPS recording with better performance if I'm saving to the internal drive it is EXTREMELY fast, I just feel like I'm putting a lot of un-needed strain on hard drives with constant writes. I also have 2 2TB EVOs I can put into a JBOD external usb-c enclosure and run some tests with that. Sorry for the wordbarf. Happy new years!

    I have an idea what I'm going to wind up doing, but I'm taking some notes and I'll let you know what I wind up with successfully.
  • BenBen
    If you offload the files yourself with some tool that's external to SecuritySpy, I don't think there is a way to keep these files visible to SecuritySpy - I doubt that symlinks would work here.

    Your external drive is very fast - I would recommend using it directly as SecuritySpy's capture destination. It has plenty of speed even for continuous recording of many high-resolution cameras. Capturing directly to the drive puts no more strain on it than would capturing to the internal SSD and then copying all those files to the external drive using some other means.
  • donbvonbdonbvonb
    edited January 2021
    So what I've done is used rsync to push the files to the external drive (a lot faster using single threads) and then a ruby script called "master_delivery" which I had to modify to make compatible with Mac, but it effectively does what I'm looking to do. The files are still accessible within your program, and it seems that this solution is indeed what I am looking for.

    Unfortunately/fortunately the guys are asking me to add another 40 Cams but I don't forsee this to be a problem in terms of IOPS, but may bring me back down from the 15fps I was looking to get, but we will see. I just wanted to confirm that symlinks do work. There is a slight additional latency (<10ms) added when viewing files from external hdd array, but the whole backup process occurs while your program is running, and rsync fails beautifully when a file is still being accessed, and with the following commands, only the files that are copied seem to be replaced by their symlink equivalents. Your program doesn't seem to be affected in the slightest to the human eye (obviously a non-zero measurable amount, but I haven't gotten there yet)

    Where /Volumes/ExternalArchive/ is the mounted 28TB

    This command backs up all Video File Data to External Drive via a copy command using rsync, ignoring already created symlinks
    rsync -aP --no-links ~/SecuritySpy/Captured\ Files /Volumes/ExternalArchive/Archives/Automatic

    This command then creates/replaces originals with symlinks and backs up copies
    master_delivery -v -d ~/SecuritySpy/Captured\ Files -m /Volumes/ExternalArchive/Archives/Automatic/Captured\ Files -t symbolic_link

    There are backups of the video files created (effectively duplicates, i think manually creating this process would ignore that step, but I'll probably throw 2x4TB SSDs in a JBOD and incorporate that somehow into the backup where it makes sense (maybe have it operate as a cache since nvme->jbod ssd will perform faster, then jbod ssd->raid0 hdd will transfer without performance impact to nvme. Having a lot of fun with this, gonna buy a license soon I think my 30 days is almost up.

    I plan to do daily rsyncs and then weekly symlink replacements, and next thing to figure out is how to increase retention to the most ridiculous amount possible.

    *** This is something I was looking to accomplish on Unifi-Video's NVR solution, and I believe this technique would be very effective on their systems as well. Hope you don't mind that mention. I don't ever read rules, I just blindly agree to stuff but I mean well :)
  • BenBen
    Sounds great - thanks for reporting back with your solution, I hope this will be useful to others. It's very interesting that SecuritySpy continues to be able to access the files via the symlinks - this is not something we designed for, but it's good to know that it works well!
  • festerfester
    For me with the new M1 Mini, I ended up getting the 16 gig ram version, but the 256 gig SSD. I didn't want to pay Apple's prices for a larger internal drive, so I added a Samsung T7 1 TB external. I try to keep the internal drive use around 50% of capacity to get as much life as possible from it, and other storage is on the external drive including S.S. Working well so far.
Sign In or Register to comment.

© Vanilla Keystone Theme 2024