Users Share Virtualization Pitfalls

It's not as ubiquitous in the data center as, say, fluorescent lights, but virtualization is closing the gap quickly.

And what can't be virtualized these days? Applications get virtualized via service-oriented architecture (SOA) that can turn any desktop into a Citrix-like terminal. Virtual file servers are streamlining workgroup operations. And storage vendors still make a half-hearted attempt to sell the benefits of virtualized LUNs, even if customers can't come up with the same ROI figures as their VARs.

But let's look more closely at virtualized servers, or virtual machines (VMs) carved out of physical servers. In less than five years, VMs have proven themselves as a way to contain runaway server growth, while actually reducing the number of servers in a data center. It wasn't hard to see that most servers were running at well under 20 percent of capacity, leaving lots of idle or underused processing power. Then there's been a steady increase in power and cooling costs, and either smaller IT budgets or more controls where new spending was concerned. IT has had plenty of incentives to give VMs a whirl.

EMC subsidiary VMware remains the clear leader in this juggernaut of a market, commanding well above 80 percent of the virtual server business, according to most analyst estimates. How long it can fend off would-be rivals like IBM, Hewlett-Packard, and Microsoft -- in addition to smaller vendors like SWsoft or Xensource -- is anybody's guess. But despite what is likely to be a softer market for IT spending in 2008, Goldman Sachs pegs server virtualization as IT's second-largest spending priority after business intelligence.

The question of whether it's a good idea to virtualize servers has been settled. What data center professionals and storage experts are grappling with now is how best to deploy and manage VM technology. Byte and Switch spoke with users in a variety of industry sectors to get a feel for where the pitfalls are, and what customers wished they knew before they unplugged that first obsolete departmental server.Next page

Warner Bros. Entertainment Inc. has about three years' virtualization experience under its belt in both Intel and Unix environments. The entertainment conglomerate was forced to consolidate servers in the summer of 2004 when temperatures in its data center rose above the temperature outside its Burbank, Calif., headquarters. "We were frying the equipment," recalls Harold Shapiro, technology architect at Warner Bros. He says most servers were running at about 5 percent to 10 percent of their capacity.

The company has since consolidated 400 servers. And while Shapiro thought they might virtualize everything, he found Intel-based servers don't handle the high I/O demands well. Consequently, he virtualizes only those apps that are relatively static and not transaction oriented. "VMware likes to say you can virtualize Exchange, etc., but dont do it if performance is critical," Shapiro says.

The highest hurdle for him where virtualization is concerned is the lack of Microsoft support, even though the vendor sells its own virtualization products. Customers at the annual VMworld tradeshow have bemoaned Redmond's indifference to making Active Directory or Exchange part of virtual environments.

"That doesn’t work for us, because we need Microsoft when something goes wrong," Shapiro says. The only place Warner uses virtualization is with Internet Information Services (IIS) with Web Services, and some other third-party apps that run in Windows. "You're between a rock and a hard place in virtualizing Microsoft products."So how is it he's able to virtualize these third-party apps? "I don't ask them to support VMware, but to support it if it's running in a standard OS in a virtual environment, which has a generic driver set," Shapiro says. All the drivers and potential incompatibilities are what creates so much instability in Windows; VMware has abstracted out all the software, and it's a far more stable environment, according to Shapiro.

What does he do if a problem does arise and the vendor insists the problem be recreated in a physical server environment and not the virtual one in which it originated? "I'm OK with that as long as they try to solve it first before we go to a physical environment," Shapiro says. "For those who want to go to the physical environment first, I won't do it. It’s a crock."

Too many vendors and VARs are putting this kind of virtualization exclusion verbiage into their licensing agreements, Shapiro fumes. "Virtualization's here and if they don’t support it, they don’t deserve to be in business," he says. "The guys who really need to hear this are the biggest vendors, like Microsoft."

Next page

Will Wilson, director of information systems for Guardian Management LLC in Portland, Ore., picks his virtualization spots very carefully. The technology works well and can be a godsend for many applications, but Wilson won't run the financial services company's mission-critical apps on VMs."Virtualization in its current form works well with a lot of second-tier products, but not first tier," Wilson says, after a couple of years' worth of experience with Microsoft Virtual Server software. SQL, Exchange, and high-end, resource-intensive apps pretty much need all the all hardware you can throw at them, he adds. But virtualizing something limits performance.

"You may say this Exchange Server or SQL Server operates optimally, but then you have to spend a lot of money to make it happen with all the overhead," says Chris Snow, a coordinator for IT engineering at St. Joseph's Hospital.

Your main server for virtualization can also lock up, he adds. "So unless you have a round-robin configuration, which gets expensive, all those apps are offline too. So I don’t plan on virtualizing any mission-critical apps," Snow says.

Wilson says users might consider virtualizing in a cluster for disaster recovery, but even that adds a layer of complexity. "Products have failover and redundancy anyway, so you might as well get another server in case there's a physical hardware problem," he says.

Guardian's getting along just fine running a couple of proprietary accounting programs and older software platforms in virtualized form. Virtualization works well for a property management app with a Windows 2000 database that was built long before Wilson arrived on the scene. "No one knows how to put that on another device, So we segment it and isolate it on some hardware. If it fails we bring the virtual device over," he maintains.Next page

Fair Isaac Corp., located in Minneapolis, has used VMware for almost four years, which nearly qualifies the credit bureau as an early adopter of virtualization technology. Back then, few of its business units were willing to entrust their analytics-intensive applications to something that wasn't tried and true. An internal roadshow helped win over users, recalls Gary Tierney, senior manager of technology and solutions delivery for Fair Isaac.

In addition to credit scores, the company also does lots of data analysis in the area of fraud -- click fraud, telco and credit card fraud, and so forth. It builds system models to analyze that data and create algorithms for detecting fraud.

"Initially, no one on the business side was willing to put their analytics on those virtual servers," Tierney explains. "We eventually got a few people to buy into it. They saw the cost savings and eventually it started growing organically." Fair Isaac now has 100 hosts, running VMware's ESX 2.5 and 3.1, working in both Red Hat and Windows environments.

The natural effect of server consolidation is that users end up banging on the remaining boxes a lot harder than before. "You tend to see a lot of hardware and memory failures because you're using the machinery harder," Tierney reports. "That’s fairly common."But he says the biggest challenge Fair Isaac has run into has been with how it laid out its storage. "We went into this with the assumption that the limiting factors would be memory or CPU utilization. That turned out to be untrue -- we’d load 32 VMs onto a four-processor 580 and end up not pushing the processor or memory," he says. Unfortunately, the limiting factor was the low number of I/Os per second, a common user complaint among virtualization users.

"Once we figured that out, we changed the way we laid out our disks and made our [server clusters] smaller. Now we do no more than five servers per cluster," Tierney explains.

Next page

Jared Beard recalls playing around with a VMware license almost nine years ago when he was still working at a dotcom for online education. The technology's potential was evident; the pitfalls would only become clear after he went to work for the University of Indiana in Bloomington, Ind.

"The main thing we ran into is the learning curve -- a huge learning curve," said Beard, now the associate director of information technology labs for the university's Kelley School of Business. The VMware licensing matrix he describes as "mind-numbing." In addition, troubleshooting VMs was time-consuming at first, and it took awhile to get speedier with installs. "Once you get past that, the flexibility you get back is worth the investment in time," he adds.Flexibility's important, since his department handles server access and capacity requests from students, faculty, and staff. He gets 15 to 20 requests per semester from students looking to use servers for competitions or capstone projects. VMs are easy to allocate and then shut down after the semester ends. While he and his skeletal staff used to manage 20 servers, they've consolidated down to three. The savings have been so impressive the school's dean just signed off on an order for more virtualization licenses in 2008.

So where were the virtual speed bumps? Beard says drive-space capacity planning problems are inevitable and encourages potential buyers and new adopters to give themselves plenty of time to work this out properly, so that one VM doesn't get short-changed at the expense of another.

"We were handing out way too much drive space -- 70 to 80 GBs per server with students running database apps and video, with IBM servers that had about 400 GBs, but they weren't filling the virtual drive space," he says. They chopped that down to about 40 Gigabytes of space, which has worked out well, at least for now. But two different content types are poised to put a strain on that.

"Many of our next round of projects are video, which takes a lot of storage. And there's a social networking component that business schools are tapping into for long-term case studies and projects for MBA students. So we need to anticipate those," Beard says. "Moore's Law is fantastic, but the pace that media is growing is outpacing Moore. It's insatiable."

He also points to VM backups as a potential pain point. "Backing up a VM is not an easy process -- a piece of that comes with Virtual Center [VMware's management framework], and other third parties," Beard says. "You have to manage the snapshots and full VMs, which means you have to anticipate storage space for the VMs and all their settings."He also suggests calling on the vendor for its expertise and to train staff. "Definitely get some training to figure out the backup maze," Beard advises.

Next page

Virtualization customers should take full advantage of their vendors' and VARs' knowledge base and training, echoes Snow of St. Joseph's Hospital. The Syracuse, N.Y., health care facility has 40 virtual servers with VMware's ESX 3 software. "Get the lead [project manager] to one of their five-day classes," he says.

Snow said he and his colleagues made sure they did their homework where the technology was concerned -- maybe too much. "We had a preconceived idea of what could and could not work virtually," Snow says. "But actually, there's more you can put on VMware than we originally thought -- except we’re out of room till we order more infrastructure."

The one deployment glitch St. Joseph's experienced came with a mail gateway. "We had planned to virtualize it, and at the same time, we were going to be doing an upgrade on the mail application itself," he says. When they put it on VMware, the gateway no longer worked. The newer version of the app took too many resources, and the hospital had to calculate whether it was worth the extra overhead to virtualize it. The short answer: It wasn't, at least not for something as important as email."You can take heavy apps and put them on VMware as long as they’re the only server instance running on that box," Snow says. "A second box gives you the high availability, but then you lose the well touted effects of consolidation. But you gain virtualized high availability and disaster recovery" functions, he says. For many organizations, that trade-off's not worth it.

Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.