2.5tb of data in a month… it is a lot to process – LITERALLY! In this video I run through the new QNAP rack mounted raid system, and Seagate NAS HDDs I am using in studio to make my workflow WORK.
NOTE – the LaCie Rugged raid is self powered via thunderbolt – but if using USB 3, you will need to use a power connection also.
Check out all the gear shown below:
Get Your Gear Out!
Check out my downloadable video series:
Intimate Portraiture series:
Take Control of The Light:
The Business of Photography:
Educating Tina:
Kickstart Your Photography 6 month course:
All of my travel, tours & workshops:
My SEAFOOD Channel:
Merch:
Mailing List:
Ethics and Conduct statement:
Subscribe HERE:
Subscribe for News:
Google Plus:
Community Forum:
Facebook:
Twitter:
Instagram:
Official Website:
Please check out my art nude photography books
Private Bodies:
Public Bodies:
___
Adorama:
B&H Photo:
Amazon USA:
Amazon UK:
Amazon Germany:
Australia:
I know this video post is a few years old now.
Just have a few questions. You mention you are fitting a desktop and a rack mounted NAS. Just wondering what level of raid you have setup from performance vs resilience?
Do you use something like RAID 10 in the desktop for performance and RAID 6 for resilience in the rack mount?
Also enterprise vs nas drives which do you use ? Enterprise drives seem to have a better life expectancy and build quality?
too much shit, the best storage units are archival DVD's or BLURAY discs, an EMP or strong magnetic discharge could blow the hell out the data of an HDD
It’s been 1.5 years. An update to this video would be great. Love your work!
appropriate cat related ad for our crazy cat man 😛
Matt, What Riad do you use. 5 or 10. Cheers
Mind you the maximum I/O speed is your SATA III interface at 6 Gbps (gigabit per second). A drive can only sustain speed that until its cache memory is full. After that it drops down to the media speed (aka sustained). Which is 210 MB/s (megabytes per second = 1.64 Gbps) on a 10TB IronWolf NAS drive (the pro version does a bit better). However, when you stripe your data (as in RAID 0 or its protected variants 5 and 6) across so many drives that each drive has enough time to write its buffer to magnetic media, then you can maintain the 6 Gbps. As long as you do not have concurrent I/Os to these disks.
6 drives with 256MB cache each, on RAID 6 should be able to do that.
If you were to have concurrent I/Os then I would prefer RAID 1 at some level as the OS/driver and RAID controller combo can distribute I/Os across mirror-halves.
If you have other cache buffers in the chain, these change the story. But if you have no backup power supplies through the chain, you should switch these buffers off.
Or run the risk to lose data during power failures. You will have redundancies and still have an original somewhere, but restoring a TB of data just sets the clock back.
Oh yes, have your techies looked at your network MTU size? And have you looked into the NAS OS for the SMB/CIFS version it was set to? If not, there is more speed there, potentially.
Matt, I have been using QNAP for over 7 years, 2 bays, 4 bays, 5 bays and i wouldn't buy any other brand. I back up our work box to my house for offsite. We have only had 2 issues over the years.
1st. A virus on Desktop PC encrypted all files in "Mapped Network Drives" set up on that Windows desktop. (offsite copy paid off)
2nd. My House got hit by lightning and the COAX for cable Internet was not protected, so it went from the wall to the modem then thru the network cables frying network ports in all devices except it took out the QNAP completely. I just bought another 5 Bay (newer model) popped in the drives in the exact same order and all RAID data was there.
Now you just need a fireproof, theft proof bunker for that thing.
I'm personally using a lot of rack gear – I have 3 servers and a rackmount NAS running (and my switches/routers/patchpanels are all also rack mountes, I have about 15U worth of gear in my basement at the moment). My nas is a Lenovo/EMC px12-350r which I got for a very low price – Came out of a test setup for a large company, but they ended up going with a different, larger system. I'm in the proces of upgrading my setup at the moment, looking at getting more 10GbE running, as well as a fiber uplink to my desktop.
It can always be interesting to keep an eye out for used gear. A lot of servers coming out of compute clusters from big enterprises are often not that old (2-4 years), but because of the size of their operation, for enterprises it will be cheaper to invest in newer gear instead (because of power/cooling/upgrade/… costs that don't really scale down to consumers). A lot of those servers will easily run for 10 years without a hickkup, so it can be a good way to save some money yet still get good gear.
Hopefully with the NBN cloud backup will improve and become more viable for people. For my workflow all RAW images are copied to Google Drive (unlimited edition @$10/month) and to two other locations including a NAS and an offsite storage.
With a 2Mbps upload speed, the daily sync to Google is usually completed overnight. My point really is for $10/m for UNLIMITED data storage, it's worth looking into Google Apps Unlimited.
** Disclaimer – My 'day job' is as a Cloud Storage / Backup provider.
The difference between a standard desktop HDD vsrsus a NAS HDD is not really the vibration marketing stuff but the way the heads are parked/energy saving mode and the timing of it, which can cause the RAID to be dropped. Think about it as the HDDs are getting 'out of sync'. You really need NAS-rated HDDs in NAS servers. And people, please stop using WD Green with your RAID setups.
As a full-time network engineer and part-time photog/videog whatever you do stick with iSCSI. iSCSI is an industry standard and very robust and fast. Also make sure and have a 10Gb connectivity option. It doesn't matter how many drives you have and how fast they are the SAN connectivity is the bottleneck. I do a mix of RAID 10(same volume very fast) and RAID 60(large volume but slower). REmember no matter what the sales docs say RAID 5 and 6 are going to be slow. It is the nature of their reliability. RAID 50 and 60 help this a little. RAID 10 is LIGHTNING fast but inefficient. It waste half of the storage but makes up for it in speed. Match the raid types capabilities to your needs and stick with iSCSI.
watching in 240p i feel ya . the internet is joke here
Thank you Matt for the very insightful and useful video!
i would have gone with synologie and HGST Nas drives.
Hi Matt, thanks for sharing your setup. Currently I'm running two separate Synology NAS units, 20TB each with WD Red Pro NAS drives.
I trust that you haven't gone with a single RAID volume across those 12-drives – RAID is not a backup; you ideally want to have isolated RAID arrays and you can have these rsync across as needed between Master/Slave setups (which is what I have).
It is also worth mentioning that you go with RAID 6 as that allows for up to 2-drives failing, and do note that drives can also fail when the array is recovering, and I have heard that RAID 5 setups run this risk during 'recovery' as the array rebuilding process is extremely disk intensive.
I am hoping to grab a third Synology 8-bay unit and max that out with larger NAS drives as I scale things in the future.
all greek to me…or geek.
I run an internal SSD for my lightroom catalogue, and then back that up internally to a hard drive locally, and also rotate two external drives offsite (every month or so take one away, bring the other back and update it, then repeat the next month). This means I always have three local copies of my photos in case of hard drive / PC failure, as well as a relatively up to date copy offsite in case of a disaster at my house.
This works at the moment because I run a single catalogue – I am curious to know Matt how you deal with moving bodies of work to the backup NAS and accessing them later. Do you store each shoot in individual catalogues and move entire catalogues back and forward when you need to?
Who is your target market with these videos – amateurs or professionals? I thought it was mainly amateur Nikon (originally) shooters, but I'm not sure anymore. Must be hard spending a month in Africa and returning home with terabytes of data – tough job but somebody's got to do it. Your disk setup is probably something no amateur will ever need or can afford.
Food for thought Matt, thank you. Re Ozzy broadband, how slow is slow? I'm in NZ and I've always thought we owned the bragging rights for crap upload speeds 🙂 If you only had photos, and not huge video files, to back up would you consider the cloud do-able?
What about offsite. How are you doing that. Thanks
A professional ICT engineer deemed RAID to be a nightmare. The faulty files in a situation were written onto the good drive.
I didn't think external HDDs existed in the 90s.
Hi Matt,
Thanks for your video. I'd like to elaborate regards consistency:
1. From what I see you have a 10 Gigabit Ethernet (10GE, 10GbE, or 10 GigE) NAS connected to your computers via a 1Gbps switch, which in turn is connected to your computer LAN 1Gbps NIC(network interface card). It is a bottleneck with reduced 1GBps speed. To make sure 10GE is active, all components must be 10GE certified.
2. Same goes for the network cables used between the components. CAT 5E rated cables support up to 1Gbps. In order to use 10GE speeds, one must buy CAT 6 or 6A rated cables.
Good video but no links and could you put up details when discussing equipment?
Hi Matt, have you got your Qnap NAS devices connected to your network using link aggregation? That would mean your network connection could be at least twice what it is now.
Another great video, but you need to loose the nylon jacket. Looks great, but too noisy!
Next time move your mic to in front of you instead of off to the right..
Having a good backup available is 1, but having one offsite (or in the cloud) is just as important nowadays. Its nice to have a nas, synced to another one, even offsite, but sync is also a risk. Once you're attacked by some ransomware all accessable files are encrypted. If you have everything on 1 big nas, this is a risk. If this gets synced to the other nas you're screwed.
When using a nas for backup, also make sure the other nas is not directly accessable by the actual devices, and make sure they have versioning ora similar fuction enabled. In those cases, you're still able to restore your files, even if the sync also has screwed you 'backup' nas.
Having it in the cloud or having it on a offline harddisk (or nas) which is connected manually is the best way.
I myself run daily backups to a server at my work. Those backups are synced to another datacenter with versioning (i can return to any point in the last 365 days). Lucky i have decent internet speeds to use
Good info, Matt- I'm an IT Architect for a high-volume website and I can honestly say that the cloud is your friend for reliable off-site archive. I archive everything I have to Amazon. Specifically, AWS s3. It's reliable and inexpensive and even though there's a learning curve, it is well worth it.
I currently keep 6 months on a home brew NAS (http://www.freenas.org), all NAS info is backed up to AWS s3 and anything older than 6 months old goes in AWS Glacier (Cheaper but has access restrictions as it's designed to be used for long-term archives).
That thing is cheap as hell for a 10GbitE. I'll have to show this to the IT director at work tomorrow, they're looking for a new super fast but well priced rack mount NAS.
My current backup solution is Time Machine writing to two separate drives, then I manually sync my important folders to two other drives every few days, or after a session of changes if I've been actively using the files. I keep one copy at work too that I'll swap with a fresh copy every couple of weeks, or again, whenever I've done some big changes.
Very nice video. But you know that you manage a big amount of data when your long term storage goes from RAID to LTO. Seriously, for this amount of data and long term storage you must take a look at a tape library. It provides long term storage or really big amounts of data, lets you make several independent copies and lets you move one of the copies to a secondary location easily.
very useful, thanks 🙂
Another great video Matt thank you. The most important thing is no software must be used to retrieve your data.
One day I was looking through my photos, that weren't backed up, and randomly decided to put them on an external hard drive. Several days later my laptop died. I felt lucky afterwords.
A RAID is not a backup, because It can get corrupted in other ways than just some single drives failing. A real backup is a full copy on a second machine, which is preferably at another location and offline. It's always god to keep that in mind.
What is your opinion Matt regarding building a custom PC and making it a NAS with FreeNAS over the qnap products?
I don't even know where to begin with back-up solutions at the moment. I use OSX; i have an external drive which i keep my RAW files/cataloges on. There not being backup up at the moment though. It's slightly concerning… If you have a partnership with QNAP, do you think you could get a coupon code for your viewers?
I'm still working towards getting a full-on NAS system, partly because I want to make sure I get a good one when it finally happens. Though I do photography for fun, not for work, so it isn't crucial.
For now, I have two extra 4 TB HGST NAS drives inside my computer for backups. From what I've read, HGST drives seem to be more reliable than Seagate.
Files enter the computer on my SSD drive as a staging area for fast editing, while keeping the copy on the memory card. Every week all of my files are backed up to the two 4 TB drives. When I've edited the files in the staging area, the finished files move to an accessible area on my main hard drive, and the raw files are moved to an external hard drive. This gives me three copies of everything. The time it takes for my memory cards to get full is long enough that I never delete files from the card until long after the weekly backup has happened.
Additionally, roughly once per month I manually back up my files to an additional HGST hard drive which is kept in a waterproof pouch, inside a fire box. If a big disaster happens and I lose all three hard drives in my main computer, I'm okay with potentially losing up to a month of files for such a rare occurrence. Ultimately I plan to have this copy located off-site.