Tape Backups Come To Gmail Users' Rescue

The folks at Google installed a new version of the Gmail back-end storage software recently. As sometimes happens when code moves from development to production, it broke,  and 0.02 percent of Gmail's multitudes lost access to their mailboxes. Google's techs are furiously restoring data from backup tapes while the blogosphere and twitterverse are all abuzz about how the cloud is falling. So what can we learn from this?

First, we learn that cloud services aren't fool-proof. Too many folks expect perfection, and, as Robert Burns said, "The best laid schemes o' mice an' men  Gang aft agley." For those who don't speak Gaelic, that's, go oft awry. All technology services will fail occasionally. The question you should ask before signing up for hosted e-mail is, Can the hosting provider keep my e-mail working at least as well as my current Exchange administrator can? In my experience, the average Exchange installation has a moderate-size service failure about once every 18 months. Gmail seems to be doing at least that well.  

This event affects about 40,000 users, so it's of the scale of a corrupt Exchange information store or single server crash--the kind of thing that happens every couple of years to all but the best Exchange users. Of course, smaller systems are easier to restore, so at the three-day mark, 95 percent of those Exchange problems are over. In contrast, I have friends who still didn't have their Gmail back as much as four days after the incident.

Then we learn that SLAs protect the fees you pay your cloud provider, not the assets you put in the cloud. Like all other data, your Gmail messages are protected by backups. Those of us who believe in a belt and suspenders are not just protected by Google's tape backup but also by our own backups. I use Google Apps e-mail with the Outlook connector, and my backup system backs up the PST file with my mail daily.  

Then we learned that despite the best efforts of the Tape is Dead Marching Band and Chowder Society, Google uses tape as the line of last defense. Gmail data is replicated among multiple data centers but also backed up on tape, and it's from those tapes that the data is being teased and restored.At least one member of the "tape is dead" crowd blogged that Google must have been using virtual tape because "Google probably doesn't have a giant, cavern-spanning data center hidden beneath some volcano in the Pacific that contains acres and acres of reel-to-reel tape machines spinning away day and night backing up your data." And, "In fact, EMC recently launched a technology that obsoletes tape backups entirely with Data Domain de-duplication for everything except a total meltdown."

Frankly, I'm tired of people who haven't worked with large tape systems (or tape at all) for a while and claim that "tape is too slow to live." Many of the organizations I work with have just the opposite problem: They can't keep an LTO-5 feed at 200MBytes per second or more backing up a single Windows or Linux server.

Large data users like CERN rely on massive tape libraries to hold their data. In fact, one of what Spectra Logic calls a mid-range tape library, the T680, can save data at 12TBytes per hour and hold 2 petabytes on 680 LTO-5 tapes. The biggest Data Domain box, a DD890, is only a little faster at 14TBytes per hour and would have to get 5-to-1 deduplication to hold as much data. The big libraries, like Spectra's Tfinity, hold hundreds of petabytes, and Oracle's T8500 holds an exabyte and can transfer 500-plus terabytes per hour.

So, the cloud isn't perfect, though it's better than Akbar and Jeff in IT. SLAs are good, but you still bear the risk, and tape is not dead. Good lessons all.