Disk as bottleneck with 30 cameras
Preface: Love SecuritySpy, used it for years, and really doing some cool things with HomeAssistant.
Now, I've been tinkering and trying to push my M1 Mac mini system to do 30 cameras and learning about the various issues that can arise at this scale. The cameras are 2-4MP each with a few 4K. The current issue I'm having is write speeds to disk. With this many cameras, I need to use external storage.
At first I was writing to a disk array on my Unraid server but that wasn't quite fast enough so I grabbed to old spinning disks and made a Raid 0 share via SMB and SS seemed to work pretty well. But the disks were small and the proof-of-concept worked for me. This was with SSv5.
I just bought a 1TB SSD (Samsung 860 EVO) and created a new share (not Raid since just 1 disk) pointing to this disk formatted with BTRFS. I get 108 MB/s write speeds to it from my Mac over SMB. Side note I thought I could get much faster over 1GB network but that's another battle for another day. The problem is that SS will not auto-delete when using this disk/share and so it fills up and SS crashes. I've tried all the different settings in SS with same results. The settings for the share seem to be the same as I've used before on SSv5 that worked fine with auto-delete.
So I tried an old Time Machine router that has a 1TB disk and shared to it via SMB as well, but it's old and my max write speeds are about 40 MB/s and I get errors in SS that say "too many open files in system". Not sure what that means but it seems to be unique to this configuration/share.
So I tried a 1TB USB3 SSD disk appliance from a box computer store that also only gives me 40-50 MB/s write speeds and I get errors in SS about disk too slow.
I think what I want to do is get my 1TB SSD share on Unraid working as my target since I get the best (albeit slower than expected) disk speeds. But SSv6 will not delete old video as intended. I am wondering if anyone has seen this issue. Maybe it's the BTRFS? What's strange is that I can open that share from Finder and delete files OK so I doubt it's the disk. Could there be an issue in SSv6 perhaps? I don't see what there would be since v5 worked fine on Raid 0 which I described at the top.
Looking for any wisdom from some of you with larger systems or knowledge of Unraid/SMB and SecuritySpy.
Comments
-
What did you set for SS in terms of how much disk space at which to begin deleting? This sounds like the setting is for too low a free space. I set my minimum free space to be about two days of recording space.
-
If I recall, I tried 10GB (free space option) and then tried delete after 1 day (I can get about 2-3 days on this disk) and then tried "auto" as a last resort.
-
I think I found the issue. SS thinks there's 5TB on that disk when it's only a 1TB drive... How do I troubleshoot this?
It seems to report other disks (local, USB, and SMB) correctly though.
-
Update: I reformatted the disk as xfs and still have same 5 TB reported.
-
Update: I think this has to do with public vs private shares on the server since 5TB is a 4Tb disk plus 1TB disk. I am going to try to remove all other shares access and see if this fixes it. I suspect it will.
I've been wrestling with this for several days, and as soon as I typed it out, I figured it out. Amazing how that works! Hopefully this helps someone else in the future going down this road.
-
The answer for anyone using Unraid... you need to enable Exclusive Shares which (as I understand it) turns off FUSE for the pool you want to write to, and FUSE can cause whacky things like what I describe above.
-
Interesting - we haven't come across this before. I'm sure this information will be useful for other users.
A disk with "only" 40 MB/s write speed, although certainly on the slow side, should be fine for most systems that do Motion Capture recording. Are you perhaps doing Continuous Capture recording from many cameras, hence your storage size and speed requirements are so much greater?
Is there a particular reason why you're using network-attached storage rather than USB/Thunderbolt? Networks are inherently less reliable (and usually slower), so we would normally recommend connecting external storage by USB 3 or Thunderbolt. In a good enclosure, a good-quality drive (especially an SSD) can provide excellent speed and reliability.
-
Correct, I'm working out using SS to do both motion and continuous recording for about 30 cameras - think family farm/homestead with lots of areas to keep an eye on from unexpected visitors and predators.
My Mac mini (M1) seems to be handling the load OK after some tweaks (like balancing software and hardware decoding), and since I run a home NAS it seems logical to store my (growing amounts of) video there. My network is pretty robust. My weak links are some 802.11s connected APs pushing video from some outposts.
Now I think I have my storage squared away and can add as many disks as I need.
