Silicon Valley Hospital's RX for Business Continuity

Earthquakes are a major concern. El Camino sits just three miles from the massive San Andreas fault and even closer to the McArthur fault line. The new hospital building--slated for completion by 2008, and part of a project that includes a nearly finished $5 million-plus data center--is designed to meet California's aggressive earthquake-preparedness law, which requires all newly erected health-care facilities to be seismically tolerant.

But the key element of El Camino's business-continuity strategy is the new DR site--an SBC Communications collocation facility in Irvine, Calif.--that's connected to the hospital by way of an OC-3 pipe. So far, the site includes EMC Clariion CX700 SAN switches and EMC Centera archiving servers, a Cisco Systems MDS 9509 switch, IBM X Series 346 servers, and Unisys ES7000 405 and 510 servers. The hospital is now testing out the failover process between the two data centers. "We want this to be a fine-tuned science," James says.

But a totally mirrored DR site just isn't cost-effective for the nonprofit hospital. "I can't replicate a mirrored, $15 million data center," James says. Instead, the backup facility will mirror only El Camino's mission-critical applications: PeopleSoft, Microsoft Exchange, Vocera voice communications badges and its SQL database.

The prioritized business-continuity plan means some failovers are automatic while others are manual. If, for example, a Veritas clustered server fails, it automatically moves to another of the clustered servers, either locally or at the DR site. But the process is manual if a Microsoft server fails.

"The Exchange servers are in standby mode and ready to be imaged with whatever server may go down here," James says. It takes less than an hour to get those servers back up and running, he adds.

The Veritas Cluster Server installation is part of a server-consolidation effort that will yield a $2.5 million ROI for the hospital over five years by letting El Camino buy new applications without the need for new servers as well, James says.

So if the hospital's complex network of water pipes were to burst in the event of a lateral-force earthquake, its Cisco Global Site Selector 4491 load balancer would automatically reroute an application from Server A at the data center to Server B at the Irvine site. "I call it the magic networking layer," James says.

Off the RichterThe new hospital facility features another unusual business-continuity technology: an earthquake-detection system that will run on a secure VLAN (virtual LAN) on the El Camino backbone network. Seismic Warning Systems' QuakeGuard sensors, which will tie in with hundreds of other such sensors around the state through a secure Internet connection, detect seismic activity before a full-blown quake occurs and send alerts around the QuakeGuard customers' networks. The sensor will trigger alarms in the hospital, "homing" elevators and opening their doors about a minute before a quake, depending on the epicenter's location. QuakeGuard will also alert IS team members by e-mail, pager and voicemail if it detects an earthquake on the way.

The hospital opted not to have the earthquake alarms trigger an automatic shutdown of its servers, mainly because it already has redundant, seismically braced UPS systems plus three generators.

As for other network and server equipment for the new hospital building, El Camino is reusing as much as possible. The hospital has bought new servers and additional switches, for instance, but the goal is to keep as much of the existing equipment. As James puts it, "it's akin to changing the engine in the car while driving down the road."

Bud James
El Camino Hospital

Bud James, 50, director of technical services at El Camino Hospital. Responsible for ensuring uptime of the hospital's network and systems, he's officially an employee of Eclipsys, which staffs the hospital's 40-person IT operation. He's been in IT for 25 years and with El Camino for two years.Code Brown: "Our new IS area, in spite of our vehement requests otherwise, is in the basement of the main hospital. We have pipes that run over our staff area and over a portion of the data center. My team calls it 'Code Brown' when one of the pipes springs a leak and a strange brown substance oozes over the desktops and walls."

Why disaster recovery doesn't always get immediate attention: "Pain exposure--some organizations don't really understand the true cost of downtime until they actually experience the pain of a catastrophe."

Best geek joke: "99 little bugs in the code, 99 bugs in the code. Fix one bug, compile it again, 101 little bugs in the code. 101 little bugs in the code...." [Repeat until BUGS = 0]

What my co-workers don't know about me: "Many know that I was in USAF Special Operations as a combat controller, but most don't know that I was later a cheerleader at the U.S. Air Force Academy."

Worst quake: "We were living in Los Angeles, about 10 miles south of the Northridge earthquake in 1994. After the quake, it looked like a war zone in some areas. One house had its entire side--the chimney side--fall over like some giant had pried it open from the top down."Wheels: "Saturn VUE. It's a solid, small-size SUV that gets 20 to 24 miles per gallon. Since it has a composite body, it doesn't rust when I use it to haul my scuba gear."

Favorite hangout: "Underwater at about 40 feet, tied to an anchor and meditating."

In my CD player: "The Dalai Lama's 'Advice on Dying and Living a Better Life.' Queued up behind that is Ken Blanchard's 'Whale Done: The Power of Positive Relationships.'"

Must-see TV: "'Stargate Atlantis' and 'Battlestar Galactica,' if I get the remote. Otherwise, it's 'Two and a Half Men' and 'Reba.'"

Actor who would play me in a film: "Steven Seagal."Comfort food: "Fish and chips."

Favorite team: "San Jose Sharks."

For fun: "I went skydiving last month and threw my daughter out of the plane to celebrate her 18th birthday (it was my 54th jump)."

Business continuity can be a matter of life or death for a hospital, so selling the El Camino board of directors on the need for a new disaster-recovery site in Irvine, Calif., wasn't the hard part. Rather, the challenge lay in setting up the facility with limited funds.

"Our upper-management team is made up of true visionaries who are technologically savvy--they understand the importance of business continuity and what it takes to make it a reality," says Bud James, director of technical services at the Mountain View, Calif., facility. "But I certainly have to operate within a budget, and don't have the ability to fully replicate every server in the data center."El Camino's IT group had to squeeze the backup-facility construction costs out of the $1 million it had already been allocated from the hospital's $20 million annual budget. There wasn't enough money to mirror all the hospital's applications, so the trick was to figure out which ones to prioritize. To do this, James and his team, with the help of Symantec (formerly Veritas), created a business-continuity plan containing an RTO and RPO (recovery-time and recovery-point objective) for data handled by each application. "We'll have hot failover for most mission-critical applications and do intelligent, remote provisioning with Acronis [backup and recovery software] for the rest of the apps," James says.

The applications chosen for mirroring were People Soft, Microsoft Exchange, Vocera and the hospital's SQL database.

Although El Camino Hospital outsources its IS department to Eclipsys (James' official employer), hospital departments still go directly to James' IS group for minor projects, such as putting up Web servers or wireless LANs. Projects costing more than $100,000 require board approval. The hospital is nonprofit, so it gets a small portion of its funding from local property taxes, as well as from federal and state funds. However, "most of it is from revenue based on our fees for services," James says. El Camino also has a large fund-raising foundation.

James' latest pitch: secure WLAN and LAN equipment that will go into the hospital's new medical office building, which is currently under construction. The project will require $150 million for secure wireless equipment, among other things. Wireless and wired LAN data will remain separate in the new building. "Hackers get sick, too," James says, "so you have to be constantly aware."

As if the construction of a new hospital and establishment of a disaster-recovery site weren't enough, El Camino is about to go live with a Windows replacement of its old mainframe clinical-application system.It hasn't been easy migrating to the new system, which must integrate with 150 other applications. The new Eclipsys SCM (Sunrise Clinical Manager), which tracks a patient's medical history, is based on Microsoft's .Net platform--but not all the hospital's other applications are.

"It's been a long, hard push," says Bud James, the hospital's director of technical services. "All these applications from different departments are mission-critical."

To integrate SCM with these non-.Net apps, an interface engine had to be set up. El Camino's accounting application, for instance, needed SCM's medical information to register a patient and to verify and bill insurance, so James' team turned to Eclipsys' eLink Interface Engine for real-time communication between the two systems. ELink also sits between other non-.Net apps. "One of the biggest challenges was ensuring that the eLink engine was properly clustered, because it needs to be highly available to support these applications," James says.

Meanwhile, the hospital's SQL server runs on 64-bit technology, but the apps don't, so El Camino isn't getting the full benefit of the faster processing speed. "If the application itself were running on a 64-bit system, it would operate more quickly and be able to take advantage of a larger memory-addressing space," James says.

Tell us about your network, and we may profile it in a future issue. Send e-mail to [email protected] or call (516) 562-59140