Washington University Medical School

Natural disasters can spell big trouble for IT managers and CIOs, and the upsurge in hurricanes, earthquakes, wildfires, mudslides, and other Acts of God has put many organizations worldwide to the most difficult of survival tests.

But plenty of users have met the challenge. We contacted a few to see how their lessons could benefit others who may someday encounter similar difficulties. What follows are some key pieces of wisdom from the disaster recovery trenches, as told to us by storage pros who've survived in their shops to tell the tale.

Make sure to monitor

Monitoring electrical power could have helped Kelly Carpenter, technical services manager at Washington University Medical School in St. Louis, to locate the source of a lightning-generated power outage more quickly than he did.

A couple of years ago, lightning had struck near Carpenter's site, burning out a cable linking both the medical school and its data center to the power grid. "Hours later, the lights started dropping off in certain places," notes the exec, adding that he realized what happened when he saw a 5-foot-wide soot mark on one of the building's walls.

Though the School's two UPS systems kicked into gear as soon as the grid link was knocked out, Carpenter was nonetheless forced to power down some of the equipment in his data center, which is packed with EMC Clariion devices, HDS hardware, and blade servers.Carpenter says the experience underlined the need for carefully policing the school's systems for power abnormalities. "With monitoring, we would have found the lightning quicker," he says.

Currently, he's in the process of implementing a series of sensors on Ethernet connections within his data center, which Carpenter says will alert him to sudden power surges caused by heat or lightning. This sensors are managed through HP's OpenView software or the OpenNMS network management technology. (See HP Drives Out Cost With OpenView and Opalis Supports HP OpenView.)

Hitch up with a hosting provider

For the Orleans Parish School Board in New Orleans, hurricane Katrina prompted a major reconsideration of the organization's disaster recovery plans. (See New Orleans Parish School Board.)

"We not only had a devastating storm, our power grid was devastated for months," explains Peggy Villars Abadie, executive director of the Board's IT division. "Even if you had extra generators, you couldnt get fuel to them because the city was filled with water."

This hellish scenario forced the board to re-focus its disaster recovery plan, and the organization now hosts student data at a secondary site in Baton Rouge, La. "[During Katrina], the only thing that was certain [to be intact], was what wasn't in the city, because the city was devastated."The board's other key systems have also been sent off-site: The Website and ERP system are now hosted by Tyler Technologies in Maine. "We have moved as many of our systems as possible to an ASP model," says Villars Abadie, adding that legacy financial systems are now hosted by Oracle.

Katrina also took out the board's email server when it hit, making communication extremely difficult. As a result, Villars Abadie has now opted for a hosted email solution. (See Managed Email: Who's Watching? and Email Gets More Outsourced Options.) "If our Outlook server goes down, email is hosted by MessageOne in Texas."

Villars Abadie is also working on implementing a SAN built from IBM storage blades, which will be used to store students' individual email accounts. Backup is a key factor in the plan. "Iron Mountain have told us that if a storm comes, they will take the tapes to Memphis."

Diversify with a SAN

Another organization that reappraised its storage infrastructure in recent years is the Beeville Independent School District in Texas.

Although Beeville avoided the worst of Katrina and Rita, officials realized that they could be next. The state of Texas mandated schools to refresh their disaster recovery plans in the aftermath of hurricane Rita, which came hot on the heels of Katrina in September 2005. "It's human nature to think, 'It won't happen here,' " says Nancy Jones, Beeville's assistant superintendent. "But we have seen what happened to other school districts, and we don't want to get into trouble."For Beeville's IT director Lawrence Garcia, this meant an overhaul of the district's storage systems, shifting from USB drives and direct-attached HP tape drives to a more modern Xiotech SAN. (See Weiss Selects Xiotech and Xiotech Receives Certification .) "We needed to find a way to store our data safely, recover it quickly, and prevent delays for our teachers and our administrators," he says. "In the past, it would have taken maybe an hour or longer to find a file, [but] now users can access their information securely on the fly."

Security is a key consideration for the district. Though it's 60 miles from the coast, it is still subject to hurricanes and storms. "We're tying up our data in several different ways so that we're not caught with one copy of our data on a server that will be flooded," explains Garcia.

As a result, student data is now replicated from a server in the district's admin office to the 4.8-Tbyte Xiotech Magnitude-3D system, which is located in a data center three miles away. This data is then backed up to tape, which is stored in an undisclosed offsite location.

Beeville's Garcia explains that a number of vendors were in the frame for the SAN project. "We looked at Dell and EMC, but they didn’t strike me as more manageable than Xiotech," he says, adding that he spent about $76,000 for the Magnitude system.

Put your people first

Managing servers and storage in a crisis is just one element of an effective disaster recovery plan, according to Charlie Pelton, CIO of Clearwater, Fla.-based financial firm Market Street Mortgage. (See People-Planning Critical to DR.)The exec, whose firm is located in storm-ravaged western Florida, explained that he also had to rethink his disaster recovery plans in the weeks following Katrina. "We talked to some folks after Katrina and asked them what their biggest problems were," he says, explaining that finding somewhere for staff to work was the biggest issue cited by users.

"We contracted with a third party to have a 48-person-capable trailer that can be dropped wherever we need it," says Pelton, adding that this can be used by his IT team in a disaster. "If their homes are not available, they need somewhere to work -- we can put the trailer in Orlando, Jacksonville, or Gainesville."

On the hardware side, Market Street Mortgage had already built a "hot site" for its Clearwater data center about five years ago. "It's a fully replicated site located in the central part of the state 50 feet above sea level," explains the exec.

The financial firm mirrors its SAN to the hot site, which is 100 miles away, using software from Veritas and Neverfail. (See THE SAK Grabs Neverfail, Neverfail Aids Organ Donation, and Neverfail Swims With Chickens.) "[The hot site] was purpose-built for this," says Pelton. "It has no windows and it's on a separate power grid."

The CIO, who faced a nerve-wracking 15 hurricanes in 2005, says that 2007 has thankfully been much quieter. "Now is the peak hurricane season -- we were lucky with hurricane Dean, [but] there's a little tropical wave out there [in the ocean now]."