Long-Term Care for Your Data

Organizations should have at least one IT person whose job or duties include managing archived data. This "go-to" long-term storage-manager position, however, has been the exception rather than the rule in most companies. But that could soon change. The Health Insurance Portability and Accountability Act (HIPAA) and other privacy-related government regulations are driving organizations to come up with well-orchestrated and managed archival strategies to better protect and preserve their long-term data. Your long-term storage person should work closely with your legal department, as well as with individual department heads, to determine their data-retention requirements. For instance, if the accounting department needs to hold financial data for seven years, then you'd want to set up a deletion schedule so that archived data remains manageable.

A long-term data-storage policy and procedures plan is crucial if your company develops software for internal use or for customers. Old code may be reused or studied, so it makes sense to keep it. Detailed sales records, customer data and other information can be used for analyzing trends, and archived company communication can pay off if you find yourself involved in a patent dispute (it can prove the code was written internally). On the other hand, you can save your company unnecessary legal problems and costs by deleting nonbusiness, or even some business, e-mail correspondence that isn't protected by law.

For the technology part of your plan, you need a simple way to locate and correlate data on a tape or CD. So choose a lookup system, such as a document-management system, with automated or report-based alerts that let you know when data has expired and can be deleted based on your storage policies.

In addition, make sure you set up a budget for long-term data storage, which should include the cost of converting old digital formats to newer ones, whether that task is performed in-house or is outsourced. Don't forget to factor in maintenance for machinery--like tape drives that read your older digital formats once you've converted them.

The Hard(ware) PartYour data won't have an afterlife if you can't read it a few years down the road. So choose storage hardware that will survive as long as you keep your data. Keep in mind that storage hardware can break or fail easily--the mechanisms in older tape drives, for example, suffer a variety of debilities, including damaged tape-handling rollers.

At a company where I worked several years ago, we used older IBM QIC (Quarter-Inch Cartridge) drives for data restoration on a dual, redundant-server system. All our locations (more than 100) had these drives. When we had to restore data from backup, we used tape. Once, when we were updating code across our 250 servers, nearly all the tape drives in the field failed, so we had to send each location a working drive.

The tape drives had been cooled by an internal fan that drew in air through the cartridge slot and out of the back of the system to keep the internal mechanisms and circuit boards from overheating. Some of the units had been left on, and thanks to the constant flow of air from the fans, little drift sculptures of dirt had settled on the read-write heads and tape-handling mechanisms. The rubber tape-handling rollers had hardened and cracked in many of the drives, and the only ones that worked properly were those that had been accidentally turned off. These units were discontinued; even more frustrating was the difficulty we had finding used tape units to replace the broken ones.

The reality is that storage hardware isn't treated with the same loving care as the tapes. The hardware is not always stored properly--look in a dusty closet, drawer or even under your desk, and you might find the remains of an older tape-backup system. Therefore, when you plan for long-term storage, take into consideration the condition of the hardware you'll be using to retrieve your data. Make sure it's stored away according to the manufacturers' specifications for temperature and other environmental conditions. And, if you've had to replace a specific tape-drive system that has failed, don't leave your long-term data on that drive format. Storage is typically arranged in tiers based on the importance of the data and the speed of access and recovery. Long-term storage sits at the bottom layers of the corporate data-aging stack.

The data stack looks something like this: First is mainline storage, where your primary storage systems, SANs (storage-area networks) and the fastest, most reliable storage reside. This is the best place for data you need regularly and quickly, such as project files. Next is secondary storage, typically in the form of direct-attached server storage and special-purpose storage. Third is nearline storage, generally used for data aging and disk-to-disk backup. At the bottom of the stack are the tape backup drives and automation systems--the end of the line for many companies' data.Some organizations have branches or secondary storage on this last storage stack, such as CD-ROM archival systems attached to the main array. Although this may have been an acceptable practice for expanding storage in the past, it's an inefficient data-storage method today. Adding branches adds complexity, which can cause confusion and errors in the backup process. All data should instead flow down the stack.

Tape is the most commonly used media for long-term storage. The main issues with tape storage, of course, are the time it takes to retrieve data and the reliability of the hardware used to access it. If stored properly, many modern magnetic tape formats have shelf lives of up to 30 years. That's improving, too: Sony is making write-once versions of the SAIT and AIT-3 tape technologies that provide permanent archival tapes. AIT-3 WORM (write-once, read-many) is shipping now, and SAIT WORM should ship soon. These tapes don't require special drives, so you can use your existing drive hardware if it's healthy.

It's a growing trend for tape formats to be backward-compatible, at least for read compatibility. Sony's AIT-1 tapes, for instance, can be read by the newer AIT-3 and AIT-2 drives. DLT, LTO and other tape formats are also providing backward-compatibility as tape-drive manufacturing companies, such as Hewlett-Packard, IBM, Quantum, Seagate, Sony and StorageTek have been listening to their customers' cries for backward compatibility.

Another long-term storage technology is optical, or magneto-optical, WORM storage. Hewlett-Packard, Plextor, Pioneer and Toshiba are the main suppliers of optical storage media and drives. HP offers high-capacity storage systems designed with 20- to 25-year backward-compatibility life spans.

There are also CD-R and DVD-writeable storage formats. CD-R, which is a write-once, read-many media, has a shelf life of only five years. Some DVD manufacturers claim 100 years for certain kinds of DVD media. DVD and similar technologies like the emerging blue laser initiatives, with higher capacity that reaches the 20 GB to 50 GB range, have made inroads in long-term storage with DVD-writeable jukeboxes (see "Pluses and Minuses of DVD Formats," page 86).Meanwhile, beware of cobbled-together storage systems from unknown vendors that cram multiple, rewriteable DVD drives into a box. You need long-term service and support, so pin down your vendor on its upgrade and shelf-life claims. Your archival system should be something that's still readable down the line, so keep your storage equipment under a maintenance contract because once it can't be serviced anymore, your data is more or less dead. And don't sell yourself short by choosing a cheap solution for your long-term data archival system. Your long-term data is just too valuable.

Steven J. Schuchart Jr. covers storage and servers for Network Computing. Write to him at [email protected].

Post a comment or question on this story.

There is no single DVD storage standard today. Instead, there are the rival DVD+R and DVD-R write-once formats. DVD+R was developed by the DVD+RW Alliance, while the DVD Forum introduced DVD-R.

The DVD Forum (www.dvdforum.org), the official DVD standards body, has struggled to set a format standard for the industry. This has bred chaos, with rival formats and confusing "+" and "-" nomenclature. But the forum's role may not matter all that much in the end, because rogue formats continue to emerge and the market will determine the DVD standard format, anyway.

The new, optical blue-laser DVD format, meanwhile, appears to be heading down the same twisted path. There are two blue-laser formats: Advanced Optical Disk (AOD) and Blu-ray. AOD, developed by NEC and Toshiba, has a capacity of 20 GB on a single-layer disk and 40 GB on a dual-layer disk. Blu-ray (www.blu-ray.com) comes from Sony, with the backing of eight other vendors, including JVC, Panasonic, Phillips and Pioneer. It has a capacity of 27 GB on a single-sided, single-layer disk and 50 GB on a single-sided, double-layer disk. The DVD Forum is studying both Blu-ray and the competing AOD optical format.All but NEC and Toshiba are backing Blu-ray, and the industry is hoping those two vendors adopt it as well. Sony has introduced a Blu-ray recorder in Japan, but the technology won't be available in the United States until 2004, with full-scale production expected in 2005. So there's still time for the DVD Forum and blue-laser developers to settle on a single format and avert yet another DVD standards war. Storage and server technology white papers

Storage and server technology research reports

Storage and server technology books

"We're on Our Way to the Utility Data Center"

"Making the Case for Both Disk and Tape""2003 Survivor's Guide to Storage and Server Technology"