Sum Of All Virtual Fears

In March Gartner ignited the blogosphere by stating the obvious: Virtualization creates new attack opportunities. There's still lots of smoke billowing around, but only time will tell how much fire is behind it, and who's fanning the flames. Vendors of new virtualized security "appliances" clearly have a stake. But many enterprises are realizing they rushed headlong into virtualization without considering the impact on their data protection policies, so IT pros do have legitimate concerns over the amount of real estate that could be consumed by a successful attack on a hypervisor.

VIRTUALIZATION
Immersion Center

NEWS | REVIEWS | BLOGS | FORUMS TUTORIALS | STRATEGY | MOREIf you're squirming right now, the big question you want answered is: Just how risk-exposed are we today? After all, in that same report Gartner predicted that a patch-worthy hypervisor vulnerability would be discovered in a mainstream product before year's end 2008. These potential vulnerabilities fall into two broad categories: First, if you can escape a client OS and move into a host OS, you have access to the data on all the other client OSes on that machine. And, there are whole new realms of rootkits being designed to take advantage of virtualization technology.

"People have been working on breaking out of the guest OS in VMware for some time now," says Greg Shipley, CTO of Neohapsis, a Chicago security consultancy, and InformationWeek contributor. "And having a hypervisor rootkit installed would be a serious threat to any org. However, I don't see the development of the rootkit being the big challenge."

It's the process used to deploy such a rootkit that really intrigues Shipley.

"What's going to require more effort: Researching a vulnerability that allows us to break out of a guest OS and gain control of the hypervisor layer, or going after an administrator and hijacking the credentials required to install the rootkit, just like any other application? If the task was on my plate, I know which route I'd go."As for breaking out of the client image, Intelguardians, a consulting company, demonstrated just such an incursion into the host OS at last month's SANSFire show. Details of the vulnerability aren't public, so it's impossible to know what the attack was successful against, but you can bet these researchers aren't the only ones in this race.The lesson is that organizations now need to assume that a sufficiently motivated attacker is capable of such an exploit, and plan accordingly. Defense in depth and proper VM layout and design, including not mixing VMs with different security postures and requirements on the same host system, are crucial.

To find out how prepared our readers are, we fielded a survey—and got some eye-popping results. We can't help thinking that the 43% saying they feel virtualized machines are just as safe and secure as traditional environments are whistling past the graveyard. Of the 384 IT operations and security professionals responding, a mere 11% have put formal strategies in place to protect their VMs.

Now, many say they're relying on their current IT policies and toolsets to manage and protect virtual servers, and that makes sense to a point. Virtualized environments do face the same operational threats and risks as traditional servers, but there are added gotchas, from intra-host threats to vetting third-party hypervisor driver add-ons to new checklist items for corporate information security policies.

Let's face it: If a traditional 1U server is compromised, you'll feel some amount of personal shame, regroup, assess damages, fix the problem and move on. Most shops have strategies in place to localize internal damage, with secondary and tertiary lines of defense to safeguard against a cascade of compromised systems. Problem is, few network monitoring and management tools are up to the task of securing guest VMs. When a traditional server gets slammed and begins displaying erratic or suspicious behavior, alarms will go off. But how effective are your tried-and-true netmon tools if all machine-to-machine communication is occurring between VMs inside your "data center in a box"? How much time will the bad guys have to probe, test and exploit intra-host weaknesses before you see what's happening?

And, is the current level of high security anxiety swirling around VM-specific environments justified? It's getting there, but that's the nice thing about smelling smoke—it warns that danger's afoot.

Continue Reading This Story...

Firm Foundation

To weigh theoretical risks as well as where new applications of old attack points are feasible, you need to understand the underlying design of virtualized hosts. Virtualization creates an abstraction layer separating guest OSes from underlying hardware, enabling multiple VMs to be hosted on a single server. Virtual machines may rely on trim hypervisors using small, privileged code bases as the foundation for this abstraction; the strength of this approach is that performance of hosted apps can reach near-native levels. Products targeting the enterprise server market, including VMware ESX, Intel VPro, Virtual Iron and XenEnterprise, favor a hypervisor design.Alternately, desktop VMs and Microsoft's virtual server offerings use a traditional "fat OS" model, where guest VMs ride atop full-fledged hosting OSes.

While hypervisors provide optimized performance and a reduced attack surface, they also bring new vulnerabilities to the party and so need to have security baked in from the beginning rather than added as an afterthought. The million-dollar question here: Is it safer to rely on the open-source community to vet and test Xen, or are VMware and other vendors of proprietary hypervisors the best path to secure hosts?

"From what I've seen, VMware's QA is pretty darn good," Shipley says. "They look like rock stars compared with many other companies. How many patches has Oracle come out with this year? I lost track as they approached triple digits."Meanwhile, XenEnterprise's upcoming 4.0 hypervisor will weigh in at a trim 60,000 or so lines of code, says Simon Crosby, XenSource's CTO. Less code equals fewer potential bugs. Moreover, XenSource, which was recently acquired by Citrix, uses IBM's secure hypervisor technology, and XenEnterprise has endured the pokes and prods of the open-source community, earning a Common Criteria Level 5 rating.

Chip designers and VM software vendors are also working to stay on top of the security struggle. Steve Grobman, Intel's Director of Business-Client architecture, says Intel VT-X server and desktop virtualization offerings are designed from the ground up to strengthen security. For example, Intel's current VT-enhanced server chipsets offer three new layers of code privilege for virtualization on top of the traditional three layers of CPU code privilege.

Of course, VMware currently owns the enterprise virtualization market, and the company is feeling pretty secure.

"Design, testing and implementation of VMware ESX server contrasts with traditional, larger-platform operating systems," says Mendel Rosenblum, VMware co-founder and chief scientist. "VMware has been focused on security concerns from our first line of code. I am 100 percent confident that we will not have a hypervisor compromise due to a design flaw."

We certainly hope his certitude proves warranted, and indeed, vendors have been successful thus far.That noise was us knocking on wood.Doomsday Time

The worst-case scenario in a hypervisor-based hosted environment? Hyperjacking, where an exploit leads to a compromised platform, allowing criminals full access to all hosted guests on a given machine. In subverting the hypervisor, malicious software could easily disguise its presence from traditional security tools that reside in either hosted-OS partitions or on any software layer above the hypervisor.

The exploit situation is analogous to the threat of cloaked rootkits compromising a standalone server OS. If you own the hypervisor, you own all data traversing the hypervisor and are in a position to sample, redirect or spoof anything you please. Without some form of failsafe, guest OSes would have no way of knowing they're running on a compromised platform.

This is the stuff of nightmares when you're talking large-scale virtualization platforms that offer 10, 50, even hundreds of hosted servers running on a single piece of hardware. The potential risk for loss of control and revenue is enormous.

The answer is to maintain the integrity of the hypervisor while building in multiple fail-safes so hosted OSes can ensure they're communicating with an untainted hypervisor as a bridge to the underlying hardware and external connections. To run an unmodified OS outside Ring 0, the hypervisor must intercept "forbidden" Ring 0 instructions and emulate them elsewhere—without the guest OS recognizing what's going on. Silicon makers are looking to help here; for example, newer Intel and AMD chips targeting the virtualization market are able to insert a new privilege level beneath Ring 0. Both provide new machine code instructions that work only at Ring -1, intended to be managed by a hypervisor. In this way, a guest OS doesn't have to be modified, and the performance penalty from emulation is reduced.It's the hypervisor's job to convince each guest OS that it and it alone has access to the host server's physical resources, while juggling access to ensure that programs and data don't leak between OSes. Additional layers of code privilege for virtualized platforms on modern chipsets allow vendors to reduce the impact of a misbehaving guest OS in the event of a security breach or errant application.

To further minimize the risk of a compromised platform intercepting guest communications to the underlying hardware, some form of transaction confirmation needs to be implemented.

The Trusted Computing Group's most widely adopted standard is TPM, or Trusted Platform Module. The TPM is a critical element for a trusted hypervisor, providing hardware-based trustable root certificates, a trusted location for performing measurements and several registries where trust measurements can be stored. TPM hardware encryption provides a guaranteed method for guest OSes to vet communications with the hypervisor.

The goal of TPM is to provide tamper detection and prevention; Intel's implementation, for example, offers trusted VMM whitelists on a hosted platform. TPM is enabled before any software is loaded and can provide owner confidence over the boot sequence and ensure the authenticity of each system element as it loads. In a nutshell, the TPM hands control of the platform to the hypervisor only after the hypervisor has been loaded into a known, trusted state.

These concepts sound familiar? In higher-end versions of Vista, Microsoft relies on chipset-based TPM to provide BitLocker functionality for encrypting data stored on local drives. Future Intel and AMD hardware platforms are also slated to use TPM to forge trusted paths to attached peripherals, relying on it to create and store unique keys for hardware-level encryption of data paths. This encryption, in combination with validation of virtualization components, should make intercepting the TPM/hypervisor handoff more difficult, increasing IT's confidence that OS communications to and from the hypervisor are untainted.The Simple Things In LifeLike a person who obsesses about being hit by lighting while driving without a seatbelt, it's the mundane dangers associated with virtualization that are most likely to bite you. For example, both fat and hypervisor-based hosts are at risk of having a guest OS compromised via traditional threat vectors and exploits. An unpatched or poorly protected public-facing server is at risk, period, whether it's running on a standalone box or as one of many VMs on a large hosted platform.

However, common sense dictates that an organization's exposure increases in tandem with its reliance on virtualization and server consolidation—the more VMs per platform, the greater the danger of an undetected intra-host problem spreading. For all practical purposes, intra-host threats are invisible to traditional off-box safeguards. External firewalls and other security tools cannot inspect or control intra-host traffic, where packets never leave the host to traverse wired infrastructure. Concerns common in the real world that are tough to catch in a complex hosted environment include extraneous or suspicious intra-host cross talk posing as legitimate traffic, which is indicative of port scans, virus behavior or other malware, and direct (targeted) or incidental denial of service attacks impacting other guest VMs due to consumption of CPU cycles, I/O resources or virtualized network bandwidth.

"The 'more eggs in one basket' risk, from a pure operational perspective, has less to do with evolving threat vectors than simply being no-duh IT," Shipley says, adding that IT groups saw the same dynamic with early SANs. "Most organizations can manage this risk by designing for extra capacity, running through virtual server migration drills and keeping up with patching."

The latter is a lesson worth repeating.

"Even though I think VMware has done a good job in reducing the attack surface, ESX/VI3 still an operating system, derived from Linux, and as such it needs to be patched," he says. "Problem is, patching ESX servers is a riskier and more intrusive proposition because you're not just taking down one OS, you're taking down all the OSes it hosts, too."The good news is that we've thus far seen relatively few critical VMware patches.Living In A Virtual World

For now, we're all biding our time, waiting on the first successful compromise of a production hypervisor or VMM. To make sure your network doesn't become a poster child, architect your host implementation to keep the potential attack surface as small as possible. Find out where third-party device drivers reside—within the hypervisor to improve performance or at a higher layer, taking a slight hit while reducing the security risk.

Disable unnecessary emulated devices and lock down extraneous features and unused services on both the host platform and guests. Remember: A virtualized machine is still a machine. While that may seem obvious, IT needs to approach VMs with the same diligence and care offered to traditional servers, including adherence to security policies and guidelines. Thirty-five percent of our survey respondents admit to having no IT security or protection plan in place, with 23% 'fessing up that their policies are works in progress. Considering that upwards of 70% of respondents have deployed at least one host platform, it's clear that unpatched or unprotected virtualized servers represent vulnerabilities just waiting to be exploited.

Ensure that security concerns, permissions and environmental settings are properly configured to follow VMs to new hosts—while environmental flexibility is a key advantage of enterprise-class offerings like VMware ESX, without proper planning the ability to move VMs on the fly can be a curse.

"Along the lines of reducing attack surfaces and general exposure, I've seen organization moving their VMware management segments off from the rest of the network and restricting who and what can gain access," Shipley says. "Clearly, firewalls in the data center is a newer trend, but certainly not one that's being driven solely by VMware. The more progressive IT teams I've seen are really starting to think about the concept of 'least privilege' models when it comes to network segmentation, and organizations can reduce their risk profiles by being diligent about restricting access to the VMware management infrastructure"Shipley also stresses that IT should never put a virtualization host machine in a position where it has to enforce network zones, for example, having an ESX parent host guest VMs in and out of a DMZ.

Can You Buy Your Way Safe?

Innovative vendors of dedicated virtualized security appliances, including Reflex's VSA and Blue Lane's VirtualShield, are focusing on virtualization as a solution to security problems, rather than just another attack vector. While we prefer to reserve the "appliance" moniker for things with three-prong plugs, we realize we're fighting a losing battle. VMware is pushing the concept of "drop in" dedicated VMs that are purpose-built and preconfigured to address specific security or management needs, and many others are rushing to this nascent space; while no big player has given formal notice, we predict traditional security vendors such as Symantec will soon enter this market in force with tailored offerings.

But is that a good thing for IT?

"I can't help but wonder if some vendors are simply looking at all the virtualization going on and saying 'Hey, how do I sell security to all these VMware shops?'" Shipley says. "Part of the burden on users/consumers of the technology is to discuss what the true threat vectors are in their networks, and then look to tools."Finally, expect VMs to take a growing role on the desktop, with notebooks and PCs designed from the ground up to support admin-locked VM partitions constantly monitoring all running areas, safely removed from user-installed malware or human error compromises.

"Intel's virtualization roadmap creates significant opportunities to combat security threats in radical new ways," says Grobman, adding that Intel has a focus on security technologies that will help ensure VMM and virtual machine integrity through hardware protections.

Finally, work to raise awareness. The last question in our reader survey was open-ended, asking if readers had additional concerns or opinions on virtualization security. Sure, we got the expected rants or raves for or against specific vendors, but a recurring theme was, "I didn't have any concerns until I completed this survey." If knowledge is power, consider yourself armed.

Google Gets a Word In

Tavis Ormandy, a Google intern working on security issues in virtualized environments, stirred things up this past spring at CanSecWest in Vancouver. In his research paper, catchily titled "An Empirical Study into the Security Exposure to Hosts of Hostile Virtualized Environments," Ormandy detailed a long list of successful exploits against commercial and open-source virtualization products. While they were "inside-out" vulnerabilities—attacks launched from within a guest OS against the VM host—he discovered flaws on every platform tested, and one of the real-world outcomes of his research was a production patch to VMware's ESX Server resolving two potential denial-of-service flaws and addressing other concerns.