External storage isn't perfect.

[EOSThree]

This post kind of fits with Cara Rose's post of a week or so ago. She had a disaster because of a lack of backing up, I nearly had one in spite of backing up. I set up an external drive network a couple of years ago to keep my photos and other important stuff on. I have used an older Simple Tech drive since about 2000 to keep data on, but I was never too serious until recently. I chose 2 Maxtor One Touch 500gb external drives and a little later a Western Digital 1TB My Book Drive.

One Maxtor failed after about a month of use, Seagate(Maxtor)replaced the drive quickly and without question. Now after about a year, the other Maxtor drive will access fine, but after about 10 minutes it will suddenly eject and not come back to life until I unplug it and re plug it in. Then it will work to a point of failure after some minutes use. The WD drive has been very reliable, so has the Simple Tech, until now. Now it won't be recognized either, and I think the data on it is lost.

I was trusting of the Simple Tech drive and was starting to believe in the Maxtors again. Now I am down to 1 drive with my system backup on it and one with my photos. On top of this somehow the sync program I was using didn't transfer 2003 to the surviving Maxtor. Luckily by transferring monthly in 2003 I was able to transfer these files to the other Maxtor.

I bit the bullet tonight and bought a Newer Technologies 2 TB x 2 TB raid 1 system with removable HDDs and USB 2, 1, eSata, and Firewire 800 access. I hope these drives will remain somewhat reliable. The raid effectively makes them one 2TB drive, but if one fails it will alert me and I will replace the damaged drive, then I can rebuild the drive from the remaining drive. Hopefully this solution will last better than the 3 drive solution I had.

[Greg McCary]

There is no perfect solution to it all. I have three drives I back up on but in the event of a total disaster my Final Edited files are stored at my web site which holds full sized images. But even that could fail. Let us know how your new system works.

[Frog]

The first western digital I had died with little use except to store.
Had to pay nearly a hundred bucks to get the data retrieved.

[SmartWombat]

I bit the bullet tonight and bought a Newer Technologies 2 TB x 2 TB raid 1 system with removable HDDs and USB 2, 1, eSata, and Firewire 800 access

Sounds like that is just mirrored.
I use Netgear NV+ with 4 drives in RAID 5.
So I get 4.5TB from 4 1.5TB drives and if any 1 of the 4 fails I can reconstruct it from the other 3. I'm planning to upgrade to 4x 2TB soon.

[Franglais]

A couple of thoughts reading through all this:

1. It is exceptionally bad luck to have several external drives fail in a couple of years. It makes me wonder if there's something else going on. The USB ports on your computer, power supply, the way you disconnect the drives, that sort of thing

2. Going to RAID-anything is not a backup solution, it just protects you from drive failure. If your system gets attacked by a virus and the files are gone then the files are gone on all the RAID disks. You have to have a backup solution which you can take offline as well as your online RAID storage

[Greg McCary]

A nice UPS is also good to have. Something to prevent power surges and damage from sudden power outages. I know we lost a few drives on our CCTV systems before we learned that one.

[BlueRob]

A couple of thoughts reading through all this:

1. It is exceptionally bad luck to have several external drives fail in a couple of years. It makes me wonder if there's something else going on. The USB ports on your computer, power supply, the way you disconnect the drives, that sort of thing

2. Going to RAID-anything is not a backup solution, it just protects you from drive failure. If your system gets attacked by a virus and the files are gone then the files are gone on all the RAID disks. You have to have a backup solution which you can take offline as well as your online RAID storage

Exactly what I was thinking Franglais is not "common" by any means the constant failure of multiple HD´s. And definitely is worth checking the other hardware that could be the cause. I have had only one HD died on me and almost sure it was of natural causes LOL, it served me many years.
For me the simpler the better...I only work with 2 or 3 HD´s one with the OS and software the other for files ant third for backup....no RAID of any sort. And always have a backup in CD´s or DVD´s. I know there are several approaches to take care of the files but having a backup in an external media like a DVD is priceless.

[freygr]

The problem with RAID arrays is that all the drives will fail about the same time! The other thing a RAID array will not survive a fire but an off site back up will.

[SmartWombat]

Which is why I'm looking at power in the garage (separate from the house) probably using solar/wind as an experiment, and extending the house LAN to there.
Then I can put redundant systems out there, like one or two of the NAS.
That covers me for house accident.

Drives failing at about the same time depends on the drives being from the same batch and being the same age, I've checked and mine aren't.
Also it assumes you don't upgrade within the MTBF lifetime of the drives.
With the current rate of technical advance, I'm going to upgrade from 1.5TB to 2TB drives before the warranty runs out - and well before the MTBF limit for power-on-hours.

Because the ReadyNAS NV+ will auto-upgrade, I can buy a drive a month (spreading the cost) and replace a 1.5TB with 2TB. Each time I do that it will rebuild the array onto the new drive automatically as it reboots.
The clever part is that once all the drives are 2TB it will then automatically resize all the partitions and give me 6TB instead of 4.5TB without any intervention.

Of course it's good to back up that data beforehand and I'll use external drives for that.
But then the 4 1.5TB drives that were part of the RAID become 4 drives that I can use for external backups!

There is a great device from Maplin (A93JJ dual drives, A83HQ single drive) that plugs into USB and you just slide in a SATA disk and it becomes and external HD without having to put it in a case.
I see it as a simple way to make backups without needing to buy a USB enclosure with each drive. Using hard disks almost as if they were very big floppies!

[EOSThree]

Here I thought backup meant a copy of a file or directory on a separate storage device. The raid array is exactly that for me, and protects me from a failing drive again. I've owned that simple tech drive for a long time without having a single problem. I think it may be just at the end of it's life.

The reason I ended up with the Maxtor drives is they were on a great sale and the Simple Tech was running out of space. I have used these drives with 3 different computers. All in the same manner. Maybe it's technique, but with my Mac; #3 computer, I alway eject the drive before I disconnect it. Before with the windows machines I just pulled the USB. All the drives were externally powered.

The first one failed almost immediately. The failure was mechanical as the drive started making nasty noises, then it would lose data, I'd re format it and it would make noise, take forever to upload anything, then, lose data again. Seagate(maxtor) repaired it without question, I don't know what they did.

I bought the WD 1 TB drive because I'd had the one failure and I'd heard more than a few bad things about the Maxtor drives, especially right before Seagate bought them. The WD hasn't been any trouble since new either. I trust it pretty well. The little Simple Tech just kept chugging along this whole time.

I've recently made a big move and of course the drives all came with me. They were packed in a box with a fair amount of padding. I am thinking this move may have been a little too much for the Simple Tech and may have done damage to the other Maxtor.

The problem sounds to be mechanical again. There is a lot of noise coming from that drive and I can access it, but if I do anything big or high speed it kicks off line.

[SmartWombat]

I thought backup meant a copy of a file or directory on a separate storage device. The raid array is exactly that for me

Not really, no.
It's used as one device, and if you delete a file - it's GONE for ever.
There is no backup!

eject the drive before I disconnect it. Before with the windows machines I just pulled the USB

Ti prevent data loss I always use the disconnect option in Windows.
Down in the system tray there is a green arrow with an anonymous grey box, hover over it and it says "Safely Remove Hardware" - it does what it says on the tin!

[Franglais]

RAID is a redundancy solution - the system survives because components are duplicated and the loss of one is not fatal. Backup implies restore - the ability to go back to a point in time where the situation was clean. RAID is not a backup solution.

You should always use the Remove Safely option with USB devices under Windows. The system caches modifications to the disk in memory and may only flush the modifications to the disk after 10 minutes or so. Windows recognises some devices (USB Keys) as being removable and writes to them immediately, but it doesn't seem to do it for all USB disks. My system sometimes refuses to release a USB disk and I have to bring the system down before removing it. I use NTFS format rather than FAT32 - the system protects the disk structure from corruption (most of the time) in the event of a system crash or abrupt removal.

Any noise from a disk is not good. If the disk makes noises as though it's working continuously then it's probably that the disk is fragmented. The data on the disk is so disorganised that the system is having to search all over the disk to find the data. I defragment my disks regularly.

When you do move you have to expect a certain amount of loss, even with professional movers. I usually count on 2% of systems lost when I manage a building move. We have replacement hardware ready, we tell the users to put their data somewhere safe and not just on the desktop hard drive, etc.

[EOSThree]

Well I didn't expect to get bagged on for my chosen method of backing up my files. And contrary to popular belief, I am backing them up, maybe not in whoever's chosen method, but my stuff is backed up to my satisfaction.

Not really, no.
It's used as one device, and if you delete a file - it's GONE for ever.
There is no backup!

Not really, no. If you sync via a contribute contribute action, then the items are added to the array, but they are never deleted unless you go in and delete them off the drive. It's a one way sync, the files only go onto the drive from the original drive.

I'll admit I may have caused the damage to the Maxtor drives by just yanking them, but the chance for the damage is in the corruption of files from removing USB without safely removing hardware.

[freygr]

Which is why I'm looking at power in the garage (separate from the house) probably using solar/wind as an experiment, and extending the house LAN to there.
Then I can put redundant systems out there, like one or two of the NAS.
That covers me for house accident.

Drives failing at about the same time depends on the drives being from the same batch and being the same age, I've checked and mine aren't.
Also it assumes you don't upgrade within the MTBF lifetime of the drives.
With the current rate of technical advance, I'm going to upgrade from 1.5TB to 2TB drives before the warranty runs out - and well before the MTBF limit for power-on-hours.

Because the ReadyNAS NV+ will auto-upgrade, I can buy a drive a month (spreading the cost) and replace a 1.5TB with 2TB. Each time I do that it will rebuild the array onto the new drive automatically as it reboots.
The clever part is that once all the drives are 2TB it will then automatically resize all the partitions and give me 6TB instead of 4.5TB without any intervention.

Of course it's good to back up that data beforehand and I'll use external drives for that.
But then the 4 1.5TB drives that were part of the RAID become 4 drives that I can use for external backups!

There is a great device from Maplin (A93JJ dual drives, A83HQ single drive) that plugs into USB and you just slide in a SATA disk and it becomes and external HD without having to put it in a case.
I see it as a simple way to make backups without needing to buy a USB enclosure with each drive. Using hard disks almost as if they were very big floppies!

Maybe you should look at unRAID server, and the nice thing it's FREE, just takes some of your time to set up.

unRAID Server 4.5-beta11

unRAID Server is a Network Attached Storage server operating system designed to boot from a USB Flash device and specifically designed for digital media Storage:

# Digital Video
# Digital Music

# Digital Images/Photos

Unique RAID System

unRAID Server employs a unique RAID technology which provides for great configuration flexibility:

# Any combination of IDE and SATA hard drives may be used.
# All the hard drives do not need to be the same size or speed.
# Hard drives not being accessed may be spun down.
# Can rebuild any single failed hard drive.

True Incremental Storage

Unlike other RAID systems, unRAID Server supports true incremental storage expansion. You can add capacity by adding more hard drives or by upgrading existing hard drives. This is a great way to make use of older, smaller hard drives you might have laying around.

For example, you might start out by installing one or two new high capacity hard drives along with some number of smaller hard drives you already own. Later, you might decide to replace one of the smaller drives, and unRAID Server will restore the data of the smaller drive onto the new drive, and then expand the file system to incorporate the full size of the new drive.

Better Fault Tolerance

Similar to other RAID systems, unRAID Server permits reconstruction of a single failed hard drive. However in the unlikely event of multiple hard drive failures, data loss would be isloated to only those hard drives which failed. In traditional RAID systems, multiple simultaneous hard drive failure results in complete data loss.

Network Attached Storage

unRAID Server is compatible with Windows Networking. Individual data disks appear as disk shares under My Network Places. In addition, unRAID Server provides a composite view of all your storage through the use of User shares.

User shares permit you to view your storage as if it were one large file system, even though each data disk has its own file system. This lets you define share names such as Movies, Video, Photos, etc. whose actual contents are spread out among multiple hard drives.

Installed On A USB Flash Storage Device

unRAID server is designed to be installed on, and boot from, a USB Flash Storage device. All configuration data is also kept on the Flash; the hard drives are only used to store user data.

Supports Multiple Hardware Platforms

unRAID Server is largely hardware independent. System builders: please visit our Hardware Compatibility page for the lastest list of supported hardware.

Similar to other RAID systems, unRAID Server permits reconstruction of a single failed hard drive. However in the unlikely event of multiple hard drive failures, data loss would be isloated to only those hard drives which failed. In traditional RAID systems, multiple simultaneous hard drive failure results in complete data loss.

Network Attached Storage

unRAID Server is compatible with Windows Networking. Individual data disks appear as disk shares under My Network Places. In addition, unRAID Server provides a composite view of all your storage through the use of User shares.

User shares permit you to view your storage as if it were one large file system, even though each data disk has it's own file system. This lets you define share names such as Movies, Video, Photos, etc. whose actual contents are spread out among multiple hard drives.

Installed On A USB Flash Storage Device

unRAID server is designed to be installed on, and boot from, a USB Flash Storage device. All configuration data is also kept on the Flash; the hard drives are only used to store user data.

Supports Multiple Hardware Platforms

unRAID Server is largely hardware independent. System builders: please visit our Hardware Compatibility page for the lastest list of supported hardware.

See website http://lime-technology.com/

See Release notes here
http://lime-technology.com/forum/index.php?topic=4674.0

Copied from link: http://linuxtracker.org/index.php?pa...17948471f89e97

[Franglais]

EOS3>> Well I didn't expect to get bagged on for my chosen method of backing up my files. And contrary to popular belief, I am backing them up, maybe not in whoever's chosen method, but my stuff is backed up to my satisfaction.

FR>> Only trying to help. Plus I may learn something (see lower)

EOS3>> If you sync via a contribute contribute action, then the items are added to the array, but they are never deleted unless you go in and delete them off the drive. It's a one way sync, the files only go onto the drive from the original drive.

FR>> You got me. I thought we were talking about RAID-1. What is a contribute-contribute action? My understanding of RAID-1 is that is works at hardware level and each block modified is written to two disks. At this level there is no concept of files - that comes higher up with the Operating System

EOS3>> I'll admit I may have caused the damage to the Maxtor drives by just yanking them, but the chance for the damage is in the corruption of files from removing USB without safely removing hardware.

FR>> There are lots of things on disk that may be corrupted by yanking it - the files themselves plus the directory structure plus the allocation table saying which blocks are allocated and which aren't. I use NTFS because most of the time the disk is intact after an incident

[draymorton]

The first western digital I had died with little use except to store.
Had to pay nearly a hundred bucks to get the data retrieved.

That's cheap. Who did it for you?

I just got an estimate of $1700 for mine...

[EOSThree]

I am probably just grumpy because I live in North Carolina now...,

I still think what I am doing is backing up, but you are right, it may not be the safest in the configuration I have now. It has me thinking a little of Carbonite or Mozy. I just think to a certain extent that will become a money pit, something that I will always have to pay for to keep my Data safe. But I guess it is worth it.

Instead of "backing up" via a simple backup, or cloning, or normal synchronization. I will synchronize in a contribute mode. You create folder pairs, one that is already on your computer, such as "my photos" or "pictures" or "my music", and then you create an identical folder on your backup device. Then you tell your synchronizer program to contribute only. This will only contribute files from your computer to the backup drive, but it won't remove the files you remove from your computer.

For example lets say you have many different folders of photos on your computer. Then lets say your hdd on your computer is getting pretty full and you would like to move some of these files to another drive to free up some space. So since you've already synchronized those files to your backup array, you just delete them from your computer HDD. The next sync won't delete them from your array, because it's only a one way street, the sync can only add(contribute)data, it won't erase data.

[Franglais]

...

Instead of "backing up" via a simple backup, or cloning, or normal synchronization. I will synchronize in a contribute mode. You create folder pairs, one that is already on your computer, such as "my photos" or "pictures" or "my music", and then you create an identical folder on your backup device. Then you tell your synchronizer program to contribute only. This will only contribute files from your computer to the backup drive, but it won't remove the files you remove from your computer.

For example lets say you have many different folders of photos on your computer. Then lets say your hdd on your computer is getting pretty full and you would like to move some of these files to another drive to free up some space. So since you've already synchronized those files to your backup array, you just delete them from your computer HDD. The next sync won't delete them from your array, because it's only a one way street, the sync can only add(contribute)data, it won't erase data.

Which synchonizer program do you use? I use the Vista Briefcase for my files from work, but that's pretty simple.

I'm looking for this sort of functionality for a Customer problem - plus it has to work on Mac plus PC, version control would be nice, etc..

[EOSThree]

The easiest one I found was on Windows, it was called MS sync toy, free from MS. I had more difficulty finding this for Mac. I am using Chronosync, it calls this action simply "backup", then it has a check box that allows you to sync deletions and another to archive replaced changes. Chronosync is pretty powerful, it can synchronize machines in a network, and has about 6 different options for synchronizing. Unfortunately, I don't know of one that will work on both OSs.

I find it pretty strange that most "back up" programs don't have this option. For a photographer it's the perfect option.

I am generally ignorant about the specifics of RAID, maybe the RAID array I purchased is simplified. I can access the drives just like I access any drive on my system. The new drive shows up as 2TB in capacity, even though it's actually two 2TB drives(4TB) that mirror. I like this because I still have a fair amount of control over what happens on all of my drives.

So right now I have my WD 1TB drive, then Newer Technologies 2TB RAID 1 array. The WD is used for "time machine" (apple's backup utility) system backup, the Newer Tech is used for photographs, music, documents, movies, etc.

[freygr]

Raid 1 (the cheapest) is a MIrror drive, it writes the same stuff to both drives.

Raid2-4 I don't know- they will have to be looked up and now are obsolete.

RAID 5 uses three or more hard drives, depending on the software all the same size. The software splits up the data and writes the data with very high level error encoding to each drive. So if one of drives of the array dies, the files can be fully recovered from the two remaining drives. The size of the file increases when this is done (all the error correction code) that is why 3 TB of hard drive in the array only reports about 2 TB of space.

[EOSThree]

Raid 1 (the cheapest) is a MIrror drive, it writes the same stuff to both drives.

Raid2-4 I don't know- they will have to be looked up and now are obsolete.

RAID 5 uses three or more hard drives, depending on the software all the same size. The software splits up the data and writes the data with very high level error encoding to each drive. So if one of drives of the array dies, the files can be fully recovered from the two remaining drives. The size of the file increases when this is done (all the error correction code) that is why 3 TB of hard drive in the array only reports about 2 TB of space.

I kind of understood that aspect of RAID, what I don't understand is that if I have a RAID 5 array, do I access the array just as if I were accessing a single drive? In other words, if I've backed up my X folder on the array, can I click on that array and access X as if I were accessing it on my HD? Or if you want to retrieve a backed up folder, do you have to use some program to access it, and it's put together as one file before you can view it.

[w.slayman]

I am not a Digital Photographer, and I have a limited knowledge of computers, I can turn one off and on, surf the web, and e-mail. That being said I would like to offer all of you the benifits of my experience.

In April of last year we had a 6.4 earthquake, and the next morning at about 1:30 my house burned to the ground, the cause was loose wires in an outlet probably caused by the earthquake I lost almost everything including my camera's, (my first, a Kodak Brownie, through my Canon A1s), my slides, negatives, and prints, the slides and negatives were in steel boxes which melted. I had thought about storing my negatives and slides in a safety deposit box, but just never got around to it.

After 18 months we have put our lives back together and replaced most of the stuff we lost, except stuff with sentimental value, like wedding rings, heirlooms, and my photographs, dating back to 1952, and my fathers and grandfathers dating back to the turn of the century (20th not 21st).

The moral of this tale is DO NOT STORE YOUR BACKUPS IN THE SAME BUILDING AS YOUR ORIGINALS.

[SmartWombat]

Yes, a RAID array looks like a single drive.
So if the RAID is drive R:\ you copy c:\photos to r:\photos

I use my RAID as the main working drive, not the single drive in the PC.
So I work slightly differently from most people, I think.

My RAID NAS goes a step further from being one drive on the PCand looks like a network server (but does more).
It can talk Apple AFS, Linux NFS, and CIFS so it works with any kind of client.
Having it separate from the computer means I can share it among all the house LAN.

[Franglais]

...
RAID 5 uses three or more hard drives, depending on the software all the same size. The software splits up the data and writes the data with very high level error encoding to each drive. So if one of drives of the array dies, the files can be fully recovered from the two remaining drives. The size of the file increases when this is done (all the error correction code) that is why 3 TB of hard drive in the array only reports about 2 TB of space.
...

RAID-5 writes data redundantly across three to five disks (if you have that many). What your operating system sees is a single disk whose size is the total of your physical disks minus one out of every five:

3 x 1TB disks = 2TB useful
5 x 1TB disks = 4TB useful
10 x 1TB disks = 8TB useful

If you lose one physical disk then the RAID system has enough information on the surviving disks to get back the missing data. When you plug in a new disk to replace the bad drive then the RAID system will rebuild the data on the new drive from the data on the others (be patient while it does this, the performance is awful).

RAID works at disk hardware level. All it sees is data blocks. It knows nothing about files, directories, file allocation tables, etc. These are all Operating System features written into data blocks on the disk. And your Operating System knows nothing about what is going on in the RAID hardware, unless it runs software that can talk to the RAID controller.