Looking for True Permanence in File Storage

I guess it had to happen. I haven't been in any IT situation yet where users on their own managed to backup everything they wanted to keep permanently. Why should my home network be any different? My son's computer boot drive running Windows XP Pro cratered and totally trashed all of his digital pictures he's taken in the last year or so. Some of them are on Debbie but he didn't save all of them on her.

The publicized reports recently of the Google and the Carnegie-Mellon hard drive studies basically reinforced my prior experiences with hard drives. Years ago in the 60's, permanent storage was considered only to be possible on paper or on paper punch cards. IBM and Univac labeled magnetic memory storage on disc, drum or tape as temporary. It seems it still is. So the question becomes, what do you use to save family pictures and videos so that they are available for later family generations to look at?

Everybody seems to think of CDs and DVDs as permanent. Are they? CD±RW and DVD±RW disks depend on laser pits recorded in layers of dye that gets “flipped” to another state to indicate a change in data state. Since I'm pretty uneducated about CDs and DVD technology, I have to do some study to find out what the materials engineers think of the “permanence” values of the various dyes used.

Kodak engineers did a study of their CDR's that uses fairly conservative methodology and they claim a lifetime of 100 to 217 years for their CDRs. But one thing stuck out, they stated that different manufacturers' CDR dyes had widely differing long-term stability. I suppose the trick will be to find out which manufacturer has the most stable dye layers.

My guess is that a CD should last longer than a DVD simply because the laser pits in the dye layer will likely be the biggest and therefore the signal to signal+noise level will be the best. Since the DVD packs more information in a smaller space, the data density is higher than a CD and the data bit size has to be smaller. This all ignores the differences of the laser light wavelengths used in the different optical disk recorders. At the same data density, blue laser light, a relatively short wavelength, used in DVDs has the ability to enable a better signal/signal+noise ratio compared to the longer red laser wavelength used for CDs. At the data density used on DVDs, there has to be some serious data signal processing used to extract reliable data.

Flash drive technology might be another choice. Yet another study area! Personal experience has shown me that flash drives are surprisingly pretty tough. I've accidently washed and dried a few USB flash drives that still work even after going for a hot ride in the electric dryer for a couple of hours. I've had one though that didn't survive its unexpected Magical Mystery Tour.

There are others looking into Digital Permanence and the big bugaboo is how long a time is defined as permanent. None of this really addresses directly the question I have about how to keep from losing permanently pictures my son thinks are cool (or whatever is the current meaning for: “Excellent”!). Even Debbie might suffer a hard drive failure induced lobotomy. As Carnegie-Mellon has shown, even hard drive Raid5 arrays are not really a viable answer.

The actual process I'll use to enforce a backup policy isn't anything more than a VB script run as a shutdown task or a scheduled task. So that part of the equation is easy enough. The other part, how to implement a reasonable storage scheme to push the data onto the selected “permanent” storage media is going to be tougher.

One possible scheme might be to use a 4 GB USB flash drive as the Samba 3 shared drive on Debbie. Have the scripts running on the Windows XP clients copy the saved files to a named folder on the shared flash drive. When the folder size gets large enough to fill a CDR, make it, verify the files and then delete them off the flash drive. Or just create a new sub-folder and rename it to match the folder name used in the scripts and keep the copied folder. When the USB flash begins to approach 95% filled, copy the entire flash off to a DVDR and empty the USB flash and start again. That will give you two opportunities to keep it for a long time on relatively static media.