Best system for archiving/storage?

[patia]

What's the best system for storing and archiving digital photos? An external hard drive, but then what?

I need my digital photo files to be organized and searchable. Right now I've got all my old photos backed up on CDs, but it is a bitch to find anything.

I use a Mac and iPhoto. Should I and can I install iPhoto on a hard drive? I would consider a switch to Aperture if somebody wanted to convince me it would be worth my while.

[Photo-John]

Patia-
First of all, CDs are bad - very bad. The plastic is the weak link and when the plastic starts to degrade, your files are hosed. Hard drives are currently the safest, most stable medium for storing data. The price per GB has come down a ton in the past couple of years so they're cost effective, too. Plus, you don't have to take the time to burn a CD, so you save time, too. The ideal setup is an external RAID system with hot-swappable drives, so when you fill one up, you can just pull it and put in a new one. There are a whole bunch of turnkey, plug-and-play solutions available now. The most popular for photographers now seems to be the Drobo system. I haven't used it but I see photographers talking about it on the Web all the time and our IT department gives it the thumbs-up, too.

For organization, I use Adobe Photoshop Lightroom. It creates a database with EXIF data, your own file structure and keywords - all searchable. It even keeps track of drives that aren't currently connected to your system. I don't have any experience with iPhoto so I can't say anything about it. My guess is that it's not enough of a power-user product to really manage a large database of photos. For that you'd probably have to step up to Aperture - or Lightroom.

Maybe someone who knows Macs can jump in and offer some more insight into iPhoto and Aperture.

[patia]

Thanks, John. I know CDs are bad. Hard drives don't strike me as ideal, either, but that's what we got right now, I guess.

What's RAID?

So you have Lightroom installed on your external hard drives?

I should have mentioned I can't spend much right now. I'm planning to buy a hard drive at Costco for $100 or less. Can't really afford new software at the moment, either, so actually I'll probably just stick with either iPhoto or just folders. I would love to install iPhoto on the hard drive if I could, so that I could browse through my archived photos just the way I do on my computer and do keyword searches. Does anyone know if that's possible?

I know iPhoto isn't designed for power users, but at this point in time, it (combined with Photoshop) suits me fine.

[Photo-John]

RAID is a multi-drive system that uses multiple drives at the same time. There are a few RAID configurations but the two main ones either use the multiple drives to increase speed, or they mirror the data so that you have two copies at all times. For our purposes you'd want a mirrored RAID setup. If one drive goes down you have a second with all of your data intact.

Software doesn't need to be installed on external drives. As long as they're just drives, you just use the software that's already on your machine. So I have Lightroom installed on my main imaging machine and my laptop and use it to view the data that's stored on my external drives. Lightroom creates its own database which you can store anywhere you want. The database doesn't need the actual photos. It's just a directory of all the photo information so that it knows where they are and can find them. I'm sure iPhoto is the same.

[readingr]

I'm going to disagree with John because you asked for an archive system.

The most rigorous things to have a backup regime that makes sure that you can recover your photos from.

Yes RAID is terrific but only if you have a good enough backup system in place to start with.

So my recommendations

Get yourself a USB drive and copy all photo's to it immediately they get put on your HDD, then copy all of the photo's to DVDs suitable for archive and store in a suitable wallet in a dark place which does not change temperature to often and preferably at you parent's or grandparent's house so that in case of fire you can still recover them.

If you have enough space on your HDD then store the USB drive when its full at your parents or just leave it disconnected and powered down when not in use. If you are running out of space then make a second USB copy as well which is used on your system when you need those photos.

Once you have all this done remember to run the disks up at least every 6 months to check them, and you should do the same with the DVDs once a year.

Once you have this level of recovery in place and have money to spare then look at the Drobo system. Excellent piece of kit.

When you buy USB drives check for MTBF figures and get the best you can afford. HDD drives do fail over time. Replace these before the MTBF runs out.

Roger R.

[Photo-John]

Thanks for posting, Roger. What is "MTBF?"

[readingr]

Thanks for posting, Roger. What is "MTBF?"

MTBF = Mean Time Between Failures
MTTF = Mean Time To Failure is sometimes quoted

So the longer the interval the better, and more expensive the disk.

Roger R.

[Medley]

Patia, Aperture is well worth looking into.

I have a Macbook Pro, iLife 09 (including iPhoto 09), Aperture2.1.4, and Photoshop CS2.

Everything that John mentioned about Lightroom, Aperture is capable of doing as well. No, Aperture does not actually install on the external drive, it simply indexes them. By doing so, you can know what's on each external drive without having to plug it in. And trust me, you're going to end up with multiple external drives that look more or less identical.

In fact, as a whole Aperture does much the same as Lightroom, and carries out major functions in a broadly similar way. Yet, in it's organizing ability, Aperture goes even further than Lightroom. It has an incomparable set of features for examining, comparing, rating, ranking, and selecting images. Most users I know don't begin to use the features to their full potential, we simply have one or two favorite ways of doing things. You could write a book on the features- in fact, people have.

Instead, let me tell you about a few of my favorite features:

Masters: Masters are equivalent to negatives in the sense that they remain stored away, untouched, until needed. If the master is "fully managed", it is stored in the library, on the computer. If it is "referenced", the mater can be stored anywhere. When you edit an image, you do not edit the master, but a version of it, and you can create as many versions as you like. Masters can be any file format, even Jpeg, but are usually Raw images.

Previews: A preview is a Jpeg image created by Aperture to represent the original master with any adjustments applied to it. Apple originally intended these previews to be created on the fly, but instead decided to store them. Doing so allows all the applications in iWork and iLife (including iPhoto) to reference them, and the Quick Preview mode loads only the previews (without loading the masters). This means that thousands of photos can be referenced very quickly, and that you can access previews for the images stored offline. The user has full control of the quality of the previews. It took me a long time to figure out why I could see images that I KNEW weren't on my hard drive.

Stacks: this is one of my favorite tools. Ever take pictures in burst mode? me too. Ever do a multiple exposure HDR image? me too. Stacks are a way of quickly organizing images within a folder. I can go to the Autostack feature, and use a slider to group all shots taken anywhere from 1/10 of a second to a full second apart. Just like that, all the images taken in burst mode are divided into their own separate stacks. Then I go to my next favorite tool.....

Compare: Compare does what a photographer would do in real life- it's displays teo images side by side, and allows you to use the zoom and Loupe features to campare the two images. Once you indicate a preference of one image over the other, the chosen image becomes the first image of the next set, and the next image in line is displayed.

It doesn't take long to get through all the stacks, and choose the best image from each and move it to the top of the stack. Add the ability to give each image keywords and ratings ( one to five stars) and you begin to understand the organizational power of Aperture.

But I also encourage you to take a look at Lightroom, because the two are very similar. I compared them before I bought, but that was a few Lightroom upgrades back- and Lightroom is worth a look. For me, it was Aperture's abilty to mesh with my Mac applications that sold me.

- Joe U.

[SmartWombat]

Stacks and Comparison are in Lightroom too, and I think they are very underused by most people.
I really like the ability to stack multiple versions of an image as you develop different ideas.
Or to stack by time so burst images are all in one stack.

They're pretty close in functionality and I think the big difference is that Aperture is Mac, where Lightroom is dual-platform Mac and PC.
As a result I think it's neither fish nor fowl, not native and not integrated into either O/S like a PC only or Mac only application would be.

[Medley]

That was pretty much the conclusion I came to a few years back too Paul. Though it seems to me now that my overall thoughts at the time were that Lightroom edged Aperture out in editing, and Aperture edged Lightroom out in organizing. Two years later though, the specifics of why that was seem a little fuzzy....

- Joe U.

[patia]

I am so sorry -- I haven't received any notifications since John's first response and didn't realize I had responses.

Thanks, everyone, for your advice. I think I will eventually upgrade to Aperture. Until I can afford that, I'll just buy a hard drive and transfer everything that's on my CDs to it. Even if it's still organized by folders, it should be a lot faster to find images on a hard drive than on CDs. And I'll be able to add to existing folders as I archive photos, instead of having to start a new CD.

I should mention I do use Time Machine and an external hard drive for weekly back-up. But I don't think I can store additional photos on it since it's formatted for Time Machine.

[Medley]

Probably best not to trust it anyway. Every time my cable goes out, The company has to zap the box, and then I get to reset everything.

Incidentally, I use the Western Digital portable external HD's with my Macbook. I've never had a problem with them, and their size (about that of a calculator) makes storing them a lot easier.

Food for thought,

- Joe U.

[drg]

As technology keeps changing, and hopefully improving, some constants need to be remembered:

Backups should be regular.
Multiple copies are a must. Here you can used DVD's/CD's for another alternative but please use high quality media. Maxell and Delkin make gold based discs (CD & DVD) that don't have the deterioration issues of the standard (cheap) types.
Off site storage. Even get a lock box/safety deposit box somewhere for those really important items.

Finally remember those environmental issues that effect everything,
Temperature, humidity, dust/dirt, shock damage - don't fold, spindle or otherwise mutilate. EMF where applicable. Ultraviolet and IR issues if the media isn't properly protected!

Then make another copy and give it to someone you know where they live!

Now what am I going to do with all these images in another ten years????

[patia]

What about online backup? Anyone use those?

[Photo-John]

What about online backup? Anyone use those?

I don't have an answer for you, Patia. But I've been curious about online backup, too. Since I can easily shoot 10 GB at one event, it just doesn't seem practical to me. But maybe someone who's used an online backup service can share some real experience with us.

[Medley]

I used Mozy (https://mozy.com/home) for awhile. It's by far the least expensive solution I found. $4.95/ month gets you unlimited backup space for one computer. Prepaying for one year gets you one month free, which means that a year's worth of unlimited backup is just $54.45 (per computer)

BUT, Mozy was NOT designed with the photographer in mind. External drives are only backed up if they are mounted at the time of backup. EVERY backup. Otherwise, they are seen as deleted files, and Mozy marks them for deletion as well (marked files are deleted by Mozy after 30 days).

What that effectively means is that you need to have every external drive mounted at every scheduled backup. It became too much of a hassle for me to deal with, so I dropped the service.

In retrospect though, a couple 1Tb drives, containing my highest rated images, might be worth looking into. I just haven't invested the timeand money into setting it up.

- Joe U.

[Photo-John]

Deleting missing drives makes that pointless. And how long would it take for it to back up a TB of data? Volume was my main concern.

[drg]

Before the Dot Com crash, there were several companies trying this model and with few exceptions they disappeared.

You have to have serious bandwidth (High speed DSL to FIOS) to even consider it.

How trust worthy is the company you are shipping your precious data off to in the WWW ether?

Lots of issues that for archiving what quickly becomes Terabytes makes it rather unwieldy and more prone to loss than not.

For backing up a netbook op/sys or a notebook file set while traveling as one more option via WiFi or the like it has a place.

An online backup depends on factors that are out of your control and thus . . . Well I not using it!

[Photo-John]

For backing up a netbook op/sys or a notebook file set while traveling as one more option via WiFi or the like it has a place.

Now that's a good idea. I hadn't thought of that. Although, if you have the storage available, and as a company I do, then just dumping it all to an FTP server would also be a good way to accomplish the same thing.

[havana_joe]

I just use folders for my archiving, in a format that is easy for me to organize. For example, PICTURES -----> PLACES -----> 2009 -----> LAS VEGAS -----> 020509. In this example, I can find all my Las Vegas pictures from my 2009 trip in just a few clicks. I may duplicate certain pictures to other folders, for example I have a FAVORITES folder that has all my top pictures in it. For storage, I use 2 external hard drives, and 2 USB flash drives. I wrote a simple batch file that I click on ("PHOTO BACKUP") which then goes into each backup drive, deletes the entire photo directory, then copies the photo directory back on to each drive. It takes a little time to run, maybe 30-35 minutes, but one click and my system does a 4 drive backup for me, and each backup now matches what is on my hard drive. Maybe it's a complicated system, but it works perfect for me.

[SmartWombat]

Now that's a good idea. I hadn't thought of that. Although, if you have the storage available, and as a company I do, then just dumping it all to an FTP server would also be a good way to accomplish the same thing.

You don't need to be a company ...
My NAS in the hall solution (three ReadyNas NV+) supports FTP, and HTTP, so unit #3 (it's only 1.5TB) is the external access box.
I use DynDNS to make it appear as smartwombat.myphotos.cc instead of just being on the house LAN.
I can FTP to it from anywhere, and provided my ADSL Modem stays up, can send as much data as I want.

NAS with RAID is not a backup solution, even with RAID5 redundancy on 4 drives it can fail, and nothing protects you from pressing the delete button!
So I buy 1.5 TB external drives when they're cheap and backup to one (or maybe two now) and put it away with its power supply! as a backup.

[readingr]

I just remembered,

I take an image of my system with the software that reads RAW with the backup so that I know I can read the RAW files if support is ever withdrawn. Been bitten once by this for a CAD format used in work which no-one thought of backing up the system and hardware to read it, making the backups useless.

Roger R.

[patia]

I found these articles:

Five Best Online Backup Tools (Lifehacker)
http://lifehacker.com/5405041/five-b...e-backup-tools

The Best Online Backup Services (PC Magazine)
http://www.pcmag.com/article2/0,2817,2288745,00.asp

I'm intrigued but I don't have broadband, so that's a stumbling block. I'd need to use WIFI and/or mail them my CDs or a hard drive. Also, I wouldn't want a fully automated system -- I'm not looking for something that just backs up my computer, but an archive for files that aren't stored on my computer. And I need to be able to browse and search for individual files at a moment's notice.

But what John said about server space made me think -- I have a website that only uses 64MB of the 10,000MB available on my GoDaddy hosting plan. I just emailed them to see if there's a way to make a folder private. (I know robots.txt will block search engines but I don't want anyone to stumble across it.) That might not work for all my photos, but it could be a great backup option.