IMAGE BACKUP! (soapbox)
Here is one of the #1 questions asked online, and BY FAR the biggest tragedy that can happen with a photographer's images- IMAGE DATA BACKUP.
If you're curious about backup, GOOD! Read on. Now. And then go back up your data. NOW!!!
If you've already been the victim of hard drive failure, and either you lost some images forever or you had to pay a hefty price to have the data recovered; well, my deepest apologies and sympathies go out to you. Maybe you should leave a comment at the bottom of this page, so that "the others" can be motivated. And again, if you're curious about how a professional photographer handles their backup workflow, read on. I'll warn you ahead of time- I'm kinda opinionated. But we can at least agree on one thing: the issue at hand is a very serious one, and you MUST back up your data somehow, otherwise you are setting yourself up for tragedy, either professional or familial...
First of all I'd like to address one thing- RAW versus JPG. No, I'm not actually going to debate the merits of either, that is an entirely different article by itself. But I want to direct my initial message to those who (for whatever reason) shoot nothing but RAW, all day every day.
If you're going to be shooting absoultely everything RAW, then there are some things you need to come to terms with:
You may have to shoot fewer pictures, or make each picture count; you know what I mean...
And/Or, you may have to concede yourself to buying FAR more hard drive storage space than most.
For example let's talk about my profession, wedding photography: If you're busting out 2000-3000 photos per wedding, and shooting 12 or 21 megapixel RAW, that's at least a 12 MB image file even if you turn RAW compression on. 20+ MB if you're shooting un-compressed or 20+ megapixels. Thats what, 30-40 GB per wedding? Then if you shoot 2-4 weddings a month, that could end up being about 1 TB per year. YIKES. Then remember you need to have (at least) two copies of your data at all times, so that's 2 TB per year, preferably more.
So, like I said, you may need to concede yourself buying tons of storage! Or, commit to shooting less, maybe just 750-1250 photos per wedding, or maybe shoot JPG some of the time, for the easy shots that are in constant light and a safe exposure, etc...
Personally, I shoot 1000-2000 images per wedding, and I shoot maybe 75% JPG, usually for the easy-pie detail, getting ready, candid etc. shots. Then I shoot RAW for the critical bridal portraits, because my style is very contrasty, pushing-the-envelope type imagery that would just be a bigger risk to shoot in JPG. Yes, I understand exposure and lighting. I nail my exposures and WB pretty well, and I know how to keep dynamic range in check using lighting equipment. But I still prefer not to take that risk. As the die-hard RAW shooters argue- memory is cheap, memories are priceless...
Okay, now as far as HOW to actually backup:
1.) I download all my cards manually, NOT using Adobe Downloader or Lightroom, or whatever. I plug the card in, double-click on it, and drag the "DCIM" folder to my desktop, then name it when it finishes downloading. I like to be in CONTROL and know exactly where my data is going.
2.) FORMAT THE CARDS IMMEDIATELY!!! I cannot stress two things enough: Firstly, DON'T WORRY, you can always recover every single image off a properly formatted card, if something terrible happens to your hard drives after download. But secondly, it is a HORRIBLE scenario to be at the next photo shoot thinking to yourself "gosh, did I back these images up yet?" First of all that's just a decision you never want to have to make, and second, once you start shooting those new images, the old images "underneath" are GONE. Only the CIA can get them back, and I don't think they offer that service to the public LOL!
Anyways that's my spiel on the initial download sequence. Like I said, DO NOT WORRY about formatting your cards immediately. It is 100% guaranteed that you can get images back if they weren't corrupted in the first place. I use PhotoRescue Pro...
3.) Then of course, you back your data up. Whatever way you can remember. Maually to DVD, to an external hard drive, or to a Drobo, etc. etc. Just get it in two places at all times, ASAP. Maybe you have an automated backup that does it every night or something. Personally I like to do that automatic backup right away. I use SuperDuper, any other Apple users should too! Time Machine works, but is NOT a long-term archive solution. The point of Time Machine is to eliminate that "oops, I didn't mean to delete that" situation where you need to go back just a couple days / weeks into your past. Or to recover recent emails etc. after your main computer drive crashes...
4.) Then as soon as time permits, back up OFF-SITE. And come on, people, be smart and mail that DVD / HD to your relative in Arizona, or wherever there are no natural disasters. NOT your office down the street that is still right on top of the San Andreas fault or, in the same hurricane / tornado death-zone. Know what I mean? Bonus points for using the internet. (Carbonite, SmugVault, etc.) ...Because then, you're not cluttering up your office with hundreds of DVDs or dozens of external hard drives... Besides, your data is much safer somewhere out there, on multiple server farms all around the country / world, than in your home office.
Okay, those are the no-compromises, MUST-DO backup procedures that you simply cannot afford to be without as a professional. Personally however I still have a few soap boxes to get up on abut a couple things-
First of all when it comes to the initial backup of ALL your images, keepers AND rejcts, well, I just don't like the idea of burning that many DVD's. DVD's are just not space-economic, especially if you have to back up 1+ TB each year like I explained above. That's 225-275 DVD's each year. I don't know how many years you plan on being in business, but in just one decade that could take up a HUGE corner of your office... Even if you took your DVD's out of the archive book and put them back on one of those giant spindles, you'd still have clutter after a few years...
Second of all, I don't really like the DROBO. It is hip and nifty, (and expandable) ...but it eliminates a couple options for you:
Firstly, since the DROBO is neither RAID 1 nor RAID 5, you cannot use it to automatically create a third backup. With RAID 1 for example you can just buy a third drive and swap out one of the perfectly good drives in the system. Because you know for SURE that each drive has ALL your data on it. Then you take that drive off-site. But with a DROBO, a removed drive literally becomes useless until it is formatted again. Which brings me to my second problem- Because the DROBO uses it's own "beyond-RAID" technology, ordinary recovery applications MAY NOT work on a crashed DROBO drive. Yes, the whole point is that if one drive crashes you're okay. But if you've got your DROBO packed with 4 drives and two drives fail at once, heaven forbid, ...you could be DEAD. Whereas with a RAID 1 system, all you may have to do is bring over that 3rd drive from off-site, put it in, and duplicate it. Or worst-case-scenario, recovery CAN be easily attempted on a failed RAID 1 drive.
Now we all know the chances and risks are VERY minor, and I'm just being paranoid, but I still urge people to consider something other than a DROBO, like a G-Safe or G-Speed.
The last soap box I have is regarding all those reject files you have permanently backed up- why? Especially if you backed up straight from your 4 GB memory card to a 4.5 GB DVD. You actually backed up RAW versions of all those out of focus shots? And you wasted 0.5 GB per DVD? This is why I do my initial backup to a hard drive and not to DVD. Then, AFTER the wedding or photo shoot is delivered and the client is satisfied, I can completely erase those reject files. I don't know about you, but for me that could be a 75% space savings. I still create a DVD archive of my critical data. (Even though my online backup of such data is probably far more secure...) If I shoot enough JPG, I can fit an entire wedding, the original "keeper" files and the JPG print files, even the JPG album spreads, onto one DVD. (Then you backup the layered album PSD spreads separately, if you choose...)
There you have it, my backup spiel. Bottom line is of course, just get your data backed up. Keeping your images on ONE computer drive, especially for years and years, is just suicide. Drives fail; it is not a matter of IF, but a matter of WHEN. End of discussion.
=Matt=
1.) I only ever use one hard drive per year really. In fact that's my New Year's resolution each and every year; to go out and buy two new hard drives for the new year! Actually I stagger them so that I don't keep all my data on two identical drives that came from the same factory at the same time, because that's just askign for trouble. What I do is, I use one of the drives to back up the entire previous year, and the other drive to start backing up the new year. Doing this each year yields two drives for each year.
2.) Don't always go by what a friend recommends! Honestly EVERY manufacturer has had it's ups and downs, it's good drives and it's bad drives. Never take advice on an entire brand in general, always go to Newegg.com and read the reviews for INDIVIDUAL models to see if there are a lot of DOA or crashed / corrupted drives in that batch.
I personally have a WD mybook that has served me well for the past two years, (it will be delegated to previous year backup next year) ...HOWEVER WD had a very, very bad reputation with it's Mybook series a while ago. The Mybook was like the iPod of external storage, and the high volume of sales affected their quality control.
3.) The one brand I *CAN* recommend is G-TECH. In general, they make good products and I've had my G-DRIVE for the longest time, (3-4 years?) and it has served me well. It's not a file backup drive though, just a bootable system backup. So I only ever turn it on every few weeks.
4.) 16 hrs is about normal, depending on the file sizes, disc RPM / cache, and the connection which I'm assuming is USB. I buy only 7200 RPM drives with a 32 MB cache, so I do about 500-800 GB in 8-12 hrs, if I remember correctly from last week's 2010 backup.
5.) Get a UPS for those external drives. The last thing you want is for a power outage to happen RIGHT when your'e transferring data! (Uninteruptible Power Supply. I use APC for mine, and it gives me 10-20+ mins of power in emergencies, or I suppose it could give me a lot more if I were *ONLY* powering an external hard drive, and I were transferring from my laptop on it's own power.
6.) Forget DROBO. DROBO is only part of a complete backup system, so if you buy one then you still need another. Here's why:
Yes, DROBO does backup your data within itself, HOWEVER it is a proprietary backup meaning as soon as something crashes and you take a drive out of the DROBO, all data on that drive is pretty much un-recoverable. Whereas with standard hard drives, and in a RAID 1 backup system, you can still access and recover data off a drive using any standard hard disk docking system. With a DROBO, all you've got is two copies of your data (hopefully) in ONE place, on ONE device, with no quick way to backup the entire set of data.
I only recommend RAID 1, and thankfully G-TECH makes the G-SAFE. RAID 1 (external, not internal) is great because you can easily swap out hard drives. Basically, what you do is buy a 3rd drive to go along with the two that already come with the unit, and whenever the RAID is done backing itself up, you just pull out one of the drives and voila, you have an off-site copy of your data! Put in the (empty) 3rd drive, and it will automatically re-duplicate itself. So now you have three copies of your data, and one of them can go reside off-site with friends or family, just in case your entire house burns down.
7.) I guess this about sums things up! The most important thing, more important than any of my other biased advice, is to get something regular, foolproof, and STICK WITH IT. Personally I buy new 1 TB hard drives each year, because I do about 1 TB of work each year. I don't need 4 TB or 8 TB of data at my fingertips, so I've never needed a DROBO or other type of device. Maybe if you shoot a quarter-million RAW 21 megapixel images per year, you might need 2 TB drives, or you might need to swap out and backup every 6 months. The bottom line is to get a system and stick with it.
8.) The other bottom line is to always have an off-site backup of any files that you can't afford to lose. Because I'm confident enough in my JPG's, I'd be happy enough with just having a JPG copy on SmugMug, I don't need to burn a copy of my RAW images to DVD or anything. HOWEVER, what do I do until I can get my JPG proofs onto SmugMug? I like to have an off-site backup of my data even before I go to bed the night of the wedding. So what I do is, after getting home from the wedding and loading my data onto the two external hard drives, I also load it onto a small portable external drive, and I just go take the dog for a walk (sometimes at 2 AM, yes) to a top secret location... (mailbox, carport, friends house, whatever works!) That way, I've got 3-4 copies of my data, and one of them is off-site, before my head even hits the pillow after shooting a wedding.
Actually I think my record is SIX copies of the data:
1.) The CF cards, because even though I format them after the downloading is done, they're still on the cards and could easily be recovered.
2.) My laptop, which I take to the wedding and often download most of the images to run a slideshow during dinner etc. SO maybe I don't have a copy of the very last images of the night, but still there's most of the other stuff.
3.) My desktop, which I only load the most recent few jobs. (only a 250 or 500 GB hard drive, to keep things running efficiently)
4 & 5.) My two external hard drives. By the way, I purchase BARE drives and just put them in a docking station, or RAID 1 array. I don't bother buying whole new external devices anymore, it's just impractical for a business that ought to be around for years and years. Bare drives are actually quite easy to store, you just gotta find somewhere safe, dry, and padded.
6.) My portable, off-site drive that goes out to somewhere at least a little bit away from the house...
Alright, there you have it! I usually charge good money for private workflow coaching! :-P
=Matt=
Tagged: backup , backup system , drobo , image backup , off-site backup , RAID1
Read MoreIf you're curious about backup, GOOD! Read on. Now. And then go back up your data. NOW!!!
If you've already been the victim of hard drive failure, and either you lost some images forever or you had to pay a hefty price to have the data recovered; well, my deepest apologies and sympathies go out to you. Maybe you should leave a comment at the bottom of this page, so that "the others" can be motivated. And again, if you're curious about how a professional photographer handles their backup workflow, read on. I'll warn you ahead of time- I'm kinda opinionated. But we can at least agree on one thing: the issue at hand is a very serious one, and you MUST back up your data somehow, otherwise you are setting yourself up for tragedy, either professional or familial...
First of all I'd like to address one thing- RAW versus JPG. No, I'm not actually going to debate the merits of either, that is an entirely different article by itself. But I want to direct my initial message to those who (for whatever reason) shoot nothing but RAW, all day every day.
If you're going to be shooting absoultely everything RAW, then there are some things you need to come to terms with:
You may have to shoot fewer pictures, or make each picture count; you know what I mean...
And/Or, you may have to concede yourself to buying FAR more hard drive storage space than most.
For example let's talk about my profession, wedding photography: If you're busting out 2000-3000 photos per wedding, and shooting 12 or 21 megapixel RAW, that's at least a 12 MB image file even if you turn RAW compression on. 20+ MB if you're shooting un-compressed or 20+ megapixels. Thats what, 30-40 GB per wedding? Then if you shoot 2-4 weddings a month, that could end up being about 1 TB per year. YIKES. Then remember you need to have (at least) two copies of your data at all times, so that's 2 TB per year, preferably more.
So, like I said, you may need to concede yourself buying tons of storage! Or, commit to shooting less, maybe just 750-1250 photos per wedding, or maybe shoot JPG some of the time, for the easy shots that are in constant light and a safe exposure, etc...
Personally, I shoot 1000-2000 images per wedding, and I shoot maybe 75% JPG, usually for the easy-pie detail, getting ready, candid etc. shots. Then I shoot RAW for the critical bridal portraits, because my style is very contrasty, pushing-the-envelope type imagery that would just be a bigger risk to shoot in JPG. Yes, I understand exposure and lighting. I nail my exposures and WB pretty well, and I know how to keep dynamic range in check using lighting equipment. But I still prefer not to take that risk. As the die-hard RAW shooters argue- memory is cheap, memories are priceless...
Okay, now as far as HOW to actually backup:
1.) I download all my cards manually, NOT using Adobe Downloader or Lightroom, or whatever. I plug the card in, double-click on it, and drag the "DCIM" folder to my desktop, then name it when it finishes downloading. I like to be in CONTROL and know exactly where my data is going.
2.) FORMAT THE CARDS IMMEDIATELY!!! I cannot stress two things enough: Firstly, DON'T WORRY, you can always recover every single image off a properly formatted card, if something terrible happens to your hard drives after download. But secondly, it is a HORRIBLE scenario to be at the next photo shoot thinking to yourself "gosh, did I back these images up yet?" First of all that's just a decision you never want to have to make, and second, once you start shooting those new images, the old images "underneath" are GONE. Only the CIA can get them back, and I don't think they offer that service to the public LOL!
Anyways that's my spiel on the initial download sequence. Like I said, DO NOT WORRY about formatting your cards immediately. It is 100% guaranteed that you can get images back if they weren't corrupted in the first place. I use PhotoRescue Pro...
3.) Then of course, you back your data up. Whatever way you can remember. Maually to DVD, to an external hard drive, or to a Drobo, etc. etc. Just get it in two places at all times, ASAP. Maybe you have an automated backup that does it every night or something. Personally I like to do that automatic backup right away. I use SuperDuper, any other Apple users should too! Time Machine works, but is NOT a long-term archive solution. The point of Time Machine is to eliminate that "oops, I didn't mean to delete that" situation where you need to go back just a couple days / weeks into your past. Or to recover recent emails etc. after your main computer drive crashes...
4.) Then as soon as time permits, back up OFF-SITE. And come on, people, be smart and mail that DVD / HD to your relative in Arizona, or wherever there are no natural disasters. NOT your office down the street that is still right on top of the San Andreas fault or, in the same hurricane / tornado death-zone. Know what I mean? Bonus points for using the internet. (Carbonite, SmugVault, etc.) ...Because then, you're not cluttering up your office with hundreds of DVDs or dozens of external hard drives... Besides, your data is much safer somewhere out there, on multiple server farms all around the country / world, than in your home office.
Okay, those are the no-compromises, MUST-DO backup procedures that you simply cannot afford to be without as a professional. Personally however I still have a few soap boxes to get up on abut a couple things-
First of all when it comes to the initial backup of ALL your images, keepers AND rejcts, well, I just don't like the idea of burning that many DVD's. DVD's are just not space-economic, especially if you have to back up 1+ TB each year like I explained above. That's 225-275 DVD's each year. I don't know how many years you plan on being in business, but in just one decade that could take up a HUGE corner of your office... Even if you took your DVD's out of the archive book and put them back on one of those giant spindles, you'd still have clutter after a few years...
Second of all, I don't really like the DROBO. It is hip and nifty, (and expandable) ...but it eliminates a couple options for you:
Firstly, since the DROBO is neither RAID 1 nor RAID 5, you cannot use it to automatically create a third backup. With RAID 1 for example you can just buy a third drive and swap out one of the perfectly good drives in the system. Because you know for SURE that each drive has ALL your data on it. Then you take that drive off-site. But with a DROBO, a removed drive literally becomes useless until it is formatted again. Which brings me to my second problem- Because the DROBO uses it's own "beyond-RAID" technology, ordinary recovery applications MAY NOT work on a crashed DROBO drive. Yes, the whole point is that if one drive crashes you're okay. But if you've got your DROBO packed with 4 drives and two drives fail at once, heaven forbid, ...you could be DEAD. Whereas with a RAID 1 system, all you may have to do is bring over that 3rd drive from off-site, put it in, and duplicate it. Or worst-case-scenario, recovery CAN be easily attempted on a failed RAID 1 drive.
Now we all know the chances and risks are VERY minor, and I'm just being paranoid, but I still urge people to consider something other than a DROBO, like a G-Safe or G-Speed.
The last soap box I have is regarding all those reject files you have permanently backed up- why? Especially if you backed up straight from your 4 GB memory card to a 4.5 GB DVD. You actually backed up RAW versions of all those out of focus shots? And you wasted 0.5 GB per DVD? This is why I do my initial backup to a hard drive and not to DVD. Then, AFTER the wedding or photo shoot is delivered and the client is satisfied, I can completely erase those reject files. I don't know about you, but for me that could be a 75% space savings. I still create a DVD archive of my critical data. (Even though my online backup of such data is probably far more secure...) If I shoot enough JPG, I can fit an entire wedding, the original "keeper" files and the JPG print files, even the JPG album spreads, onto one DVD. (Then you backup the layered album PSD spreads separately, if you choose...)
There you have it, my backup spiel. Bottom line is of course, just get your data backed up. Keeping your images on ONE computer drive, especially for years and years, is just suicide. Drives fail; it is not a matter of IF, but a matter of WHEN. End of discussion.
=Matt=
1.) I only ever use one hard drive per year really. In fact that's my New Year's resolution each and every year; to go out and buy two new hard drives for the new year! Actually I stagger them so that I don't keep all my data on two identical drives that came from the same factory at the same time, because that's just askign for trouble. What I do is, I use one of the drives to back up the entire previous year, and the other drive to start backing up the new year. Doing this each year yields two drives for each year.
2.) Don't always go by what a friend recommends! Honestly EVERY manufacturer has had it's ups and downs, it's good drives and it's bad drives. Never take advice on an entire brand in general, always go to Newegg.com and read the reviews for INDIVIDUAL models to see if there are a lot of DOA or crashed / corrupted drives in that batch.
I personally have a WD mybook that has served me well for the past two years, (it will be delegated to previous year backup next year) ...HOWEVER WD had a very, very bad reputation with it's Mybook series a while ago. The Mybook was like the iPod of external storage, and the high volume of sales affected their quality control.
3.) The one brand I *CAN* recommend is G-TECH. In general, they make good products and I've had my G-DRIVE for the longest time, (3-4 years?) and it has served me well. It's not a file backup drive though, just a bootable system backup. So I only ever turn it on every few weeks.
4.) 16 hrs is about normal, depending on the file sizes, disc RPM / cache, and the connection which I'm assuming is USB. I buy only 7200 RPM drives with a 32 MB cache, so I do about 500-800 GB in 8-12 hrs, if I remember correctly from last week's 2010 backup.
5.) Get a UPS for those external drives. The last thing you want is for a power outage to happen RIGHT when your'e transferring data! (Uninteruptible Power Supply. I use APC for mine, and it gives me 10-20+ mins of power in emergencies, or I suppose it could give me a lot more if I were *ONLY* powering an external hard drive, and I were transferring from my laptop on it's own power.
6.) Forget DROBO. DROBO is only part of a complete backup system, so if you buy one then you still need another. Here's why:
Yes, DROBO does backup your data within itself, HOWEVER it is a proprietary backup meaning as soon as something crashes and you take a drive out of the DROBO, all data on that drive is pretty much un-recoverable. Whereas with standard hard drives, and in a RAID 1 backup system, you can still access and recover data off a drive using any standard hard disk docking system. With a DROBO, all you've got is two copies of your data (hopefully) in ONE place, on ONE device, with no quick way to backup the entire set of data.
I only recommend RAID 1, and thankfully G-TECH makes the G-SAFE. RAID 1 (external, not internal) is great because you can easily swap out hard drives. Basically, what you do is buy a 3rd drive to go along with the two that already come with the unit, and whenever the RAID is done backing itself up, you just pull out one of the drives and voila, you have an off-site copy of your data! Put in the (empty) 3rd drive, and it will automatically re-duplicate itself. So now you have three copies of your data, and one of them can go reside off-site with friends or family, just in case your entire house burns down.
7.) I guess this about sums things up! The most important thing, more important than any of my other biased advice, is to get something regular, foolproof, and STICK WITH IT. Personally I buy new 1 TB hard drives each year, because I do about 1 TB of work each year. I don't need 4 TB or 8 TB of data at my fingertips, so I've never needed a DROBO or other type of device. Maybe if you shoot a quarter-million RAW 21 megapixel images per year, you might need 2 TB drives, or you might need to swap out and backup every 6 months. The bottom line is to get a system and stick with it.
8.) The other bottom line is to always have an off-site backup of any files that you can't afford to lose. Because I'm confident enough in my JPG's, I'd be happy enough with just having a JPG copy on SmugMug, I don't need to burn a copy of my RAW images to DVD or anything. HOWEVER, what do I do until I can get my JPG proofs onto SmugMug? I like to have an off-site backup of my data even before I go to bed the night of the wedding. So what I do is, after getting home from the wedding and loading my data onto the two external hard drives, I also load it onto a small portable external drive, and I just go take the dog for a walk (sometimes at 2 AM, yes) to a top secret location... (mailbox, carport, friends house, whatever works!) That way, I've got 3-4 copies of my data, and one of them is off-site, before my head even hits the pillow after shooting a wedding.
Actually I think my record is SIX copies of the data:
1.) The CF cards, because even though I format them after the downloading is done, they're still on the cards and could easily be recovered.
2.) My laptop, which I take to the wedding and often download most of the images to run a slideshow during dinner etc. SO maybe I don't have a copy of the very last images of the night, but still there's most of the other stuff.
3.) My desktop, which I only load the most recent few jobs. (only a 250 or 500 GB hard drive, to keep things running efficiently)
4 & 5.) My two external hard drives. By the way, I purchase BARE drives and just put them in a docking station, or RAID 1 array. I don't bother buying whole new external devices anymore, it's just impractical for a business that ought to be around for years and years. Bare drives are actually quite easy to store, you just gotta find somewhere safe, dry, and padded.
6.) My portable, off-site drive that goes out to somewhere at least a little bit away from the house...
Alright, there you have it! I usually charge good money for private workflow coaching! :-P
=Matt=
Tagged: backup , backup system , drobo , image backup , off-site backup , RAID1
