Baptist Health

It took Baptist Health, a not-for-profit health care organization in Montgomery, Ala., seven months to get its disaster-recovery network in place -- but now that the project is finished, it's fully prepared to remain up and running, even if one of its sites is completely destroyed.

About a year ago, the group installed Nortel Networks Corp. (NYSE/Toronto: NT) OPTera Metro 5200 DWDM (dense wave division multiplexing) equipment to link two EMC Corp. (NYSE: EMC) Symmetrix 8730s in its two primary data center sites over dark fiber. The Nortel OPTera and Symmetrix arrays are connected at each of the sites via two Brocade Communications Systems Inc. (Nasdaq: BRCD) 16-port 1-Gbit/s SilkWorm 2800 switches. The aim was to provide synchronous data mirroring between the two sites, which are 11 miles apart, to ensure data would be protected in case of a disaster.

But the process of getting the technology to all work together was... oh, we can't help it... kind of a baptism by fire.

"This solution had been tested by EMC for both the Brocade and Nortel gear, but, well, it hadn't been done in real life," says David Billingsley, the organization's chief technical architect. "It was very hard to find a reference site."

No significant technical glitches cropped up, he says. Rather, it was more a matter of "getting the terms defined among the three vendors... it was a knowledge-base issue, not a technical issue."Baptist Health, which has three hospital facilities and 11 additional office locations, got a good deal of help in putting the network together. Engineers from EMC, Brocade, Nortel, and McKesson Corp., a major supplier of health care software and consulting services, collaborated on the project. In addition, it was assisted by KMC Telecom, which provided the dark fiber.

But it still took time and effort to get the infrastructure running smoothly. "You have a plan, and there are examples in the lab, but they don't exactly mirror reality, so we had to improvise in some places," Billingsley says. "At certain points, we had to get down to figuring out the nanometer rates on the fiber."

Now, the hospital system is ready for a "smoking-hole disaster," Billingsley says -- that is, a catastrophic event that leaves nothing but a smoking hole in the ground. [Ed. note: Smoking-Hole Disaster? Wasn't that a Steven Seagal flick?]

Baptist Health uses EMC's Symmetrix Remote Data Facility (SRDF) synchronous disk mirroring software and TimeFinder local mirroring software to keep up-to-the-second copies of all its data. "The advantage is that because it's real time, I can connect the servers to the storage and be back up in an hour from something major happening," Billingsley says. "It's not just having a disaster-recovery plan in place... it's being able to recover within a specific time period."

Currently, the organization is mirroring about 2 Tbytes from one data center to another. In about a year and a half, Billingsley says, that will increase to 10 Tbytes. The main applications running on its two SANs are McKesson's STAR Patient Care, STAR Financials, and Horizon Clinical Documentation systems -- in other words, the lifeblood of the hospital system."In health care, if you lose some transactional data, that's money that's lost," Billingsley says. "There's also the clinical data: What goes into the electronic medical record goes into the patient's chart. So you've got both types of data you're trying to protect."

Another factor that spurred Baptist Health to put site-to-site mirroring into place was the Health Insurance Portability and Accountability Act (HIPAA), a law that imposes stricter data management requirements on health organizations. "The HIPAA regulations require increased security," Billingsley says. "But if you don't have the data, you can't secure it."

Before it installed the DWDM network, Baptist Health had used multiple T-1 lines between its data centers, which made site-to-site data replication a complex and time-consuming task. A DWDM network is literally several thousand times faster than the T-1 leased lines: Where a T-1 runs at a pokey 1.5 Mbit/s, the DWDM network can operate at up to 10 Gbit/s per fiber pair. Baptist Health has six fiber pairs per location (for more on this technology, see our recent report on Storage Over Optical).

DWDM lets Baptist Health run multiple protocols over the same fiber pair, including Fibre Channel, Gigabit Ethernet, and even the PBX voice traffic. Moving to its own dark fiber network also meant it didn't need to keep paying the local exchange carrier, BellSouth Corp. (NYSE: BLS), for those T-1s.

Here are before-and-after diagrams of its metro-area network configuration:

Figure 1: Source: Baptist Health

Figure 2: Source: Baptist Health

What's next? The hospital group wants to run completely disparate fiber connections among its sites so there isn't a single point of failure -- "So one backhoe doesn't kill us," explains Billingsley.

The organization is also gearing up to migrate its 9 Data General Unix (DG-UX) servers (out of its 150 servers total) to IBM Corp. (NYSE: IBM) AIX servers. It's doing this reluctantly, and only because EMC -- which bought Data General in 1999 -- has phased out development and support of the server technologies.

"DG-UX is going by the wayside, which is a shame," says Billingsley. "The way Data General Unix addresses storage and storage devices is far advanced compared with the other Unix vendors. Now that functionality is going away."By using clustering software running on IBM AIX, Baptist Health hopes to achieve "true business continuity" by clustering servers in its two data centers over the Nortel DWDM network by virtually mirroring the IP addresses of those servers. This, Billingsley notes, is an even more complex setup than mirroring data. "You can't just do the apps. You also have to consider the servers, the network, and the other applications," he says.

The good news is that out of the hospital group's experience, McKesson and EMC have developed a model about how to implement site-to-site replication at other hospitals. "Everyone tells us we're the first ones in health care to pull this off," Billingsley says. "We learned some hard lessons, but our ultimate goal is to make sure we don't run into the same issues."

Todd Spangler, US Editor, Byte and Switch