Utility Sings Virtualization's Praises

LOS ANGELES -- VMWorld -- Dave Trupkin always thought disaster would come in the form of a fire or an earthquake. As senior systems administrator for the Las Vegas Valley Water District in Nevada, he's paid to consider such prospects -- as well as the contingencies.

But "The Disaster," as he refers to it, did strike over Labor Day weekend two years ago in the form of a data center power failure -- less flashy than a natural disaster maybe, but no less dramatic in its impact on the water utilitys operations, he told an audience here today.

System power was supposed to have been restored the next day, but it took almost three days to get servers, systems, databases, and applications humming again. And that was only because Trupkin and his department were able to scare up some very tough-to-find generators, which LVVWD then relied on for eight more weeks till the electrical problem could be identified and repaired. (Overheated, parallel power lines had burned through their insulation.)

The utility was no stranger to VMware products or the upside of virtualizing servers and applications. In fact, use of the vendor's ESX Server to restore servers quickly and relatively painlessly after The Disaster "made us heroes with management," Trupkin laughed. It also cleared the way for broader use of virtualization across LVVWD's information systems and use of a collocation facility 10 miles away, hot-linked to LVVWD's data center with fiber pairs.

The Disaster pointed up the woeful inadequacy of the utility's "Tape and Pray" DR plan, and it sensitized management, IT staff, end users, and customers to the need for redundancy, smarter and faster backup, and electrical powering issues beyond LVVWD’s control."It takes six weeks to get a new circuit breaker installed for a new server," Trupkin says. "[Electrical] power is our main issue, so we had to learn to do more with less. And that led us to virtual machines," which wring maximum efficiency out of the processing power and capacity of each active server.

"We have 17 hosts now, and any new servers must be virtual, unless you can prove that there's no way to run an application in a virtual environment," Trupkin says. Application software vendors, once notorious for their pushback against virtualized apps, "have been dragged to the party kicking and screaming," he notes. Voice platform maker Avaya resisted the virtualization pressure till it saw what LVVWD could do with virtualized voice apps. "Now they push their other customers to virtualize."

An audience query, "Can you get them to call Cisco?" prompted some laughs and many nods of recognition.

Sadder but wiser (and a lot more tired) for the experience, Trupkin offered up these lessons learned:

— Terry Sweeney, Editor in Chief, Byte and Switch