The Truth About Storage Reliability

Two papers presented at the Usenix File and Storage Technology conference in February challenge many of the assumptions we've long used as the basis of our storage-system designs, most significantly the 1 million hour or higher MTBF found on the spec sheet of almost all disk drives sold for server use.

In both "Disk Failures in the Real World: What Does an MTTF of 1 Million Hours Mean to You," by Bianca Schroeder and Garth A. Gibson of Carnegie Mellon University, and "Failure Trends in a Large Disk Drive Population," by Eduardo Pinherio, Wolf-Deitrich Weber and Luis André Barroso of Google, the actual failure rate of drives was typically more than four times the 0.88 percent AFR (Annual Failure Rate) that a million-hour MTBF represents.

Each group studied the replacement history of more than 100,000 disk drives over data center lifetimes. CMU's samples included both enterprise (SCSI and Fibre Channel) and high-capacity drives with SATA interfaces. Google used desktop style ATA and SATA drives with spec-sheet MTBFs of 400,000 hours in their custom servers. Both studies used the same definition of a drive failure that you or I would use: If a drive had to be replaced by the data center maintenance team for any reason, it was declared a failed drive.

As a charter member of the "you can tell a vendor is lying if his lips are moving" club, I wasn't all that surprised that drives fail more than once every million hours. I was a bit surprised, though, by some of the studies' other findings. In the CMU study, SATA drives failed at about the same rate as the enterprise SCSI and Fibre Channel (FC) drives, contrary to the conventional wisdom that enterprise drives are 50 percent to 100 percent more reliable than their SATA counterparts.

Even more surprising was that drive-failure rates increased as drives aged, even within the five years most of us consider the reasonable service life of a disk drive, and there was no observed peak in drive failures at the beginning of the drives' lives due to infant mortality. In fact drive failures in years 4 and 5 were up to 10 times the rate predicted by the vendor spec sheets.And get this, the studies found that drive failures in individual systems come in clusters. The CMU group found that drives were significantly more likely to fail within hours when another drive in the same server cluster failed. That's bad news for RAID 5 fans, since drive capacity has grown much faster than data-transfer rates, so even in systems with hot spares, it could be several hours, or longer, before the array is rebuilt. A second drive failure, or unrecoverable read error, during that time would mean lost data.

Although SATA reliability appears to rival that of SCSI and FC, enterprise-class drives--due to their higher rotational speed and faster positioners--provide better performance in random I/O applications. SAS (Serial-Attached SCSI) and FC drives are dual-ported, giving redundant RAID controllers independent paths to the drive so a controller or data-path failure won't result in data loss.

So what can we storage managers take from all this academic research? First, choose your drives based on capacity and performance rather than vendor claims. If your application needs lots of space and will primarily do sequential I/O, go with lower cost SATA drives without feeling guilty about giving up reliability--because you haven't. Second, since the odds of multiple drives failing in an array before the array can be re-built are higher than we imagined, you should investigate RAID-6, data replication or other arrangements that can survive multiple drive failures. Now that vendors from Network Appliance to AMCC are speeding up the parity calculations on their RAID 6 controllers, I'm recommending RAID 6 for all the arrays we build with 500-GB and bigger drives.

Howard Marks is founder and chief scientist at Networks Are Our Lives, a network design and consulting firm in Hoboken, N.J. Write to him at [email protected].