3Leaf Dynamic Data Center: True Hardware Virtualization

3Leaf's Dynamic Data Center (DDC) is a rack of servers, storage and networking that does something unique—it allows administrators to treat the entire rack of systems as one computer running a single OS over multiple AMD Opteron servers. The location of underlying hardware no longer matters. Now that is virtualization. 3Leaf is positioning the DDC to sell to OEM vendors and limited fixed configurations to end-users. Systems will be available in December, 2009.

At a high level, 3Leaf's DDC product line virtualizes and aggregates computing hardware across multiple platforms. A fully racked DDC rack operates as a single computer with 192 cores and 1TB of RAM. To provision a server, you select the number of cores, the amount of RAM and the required I/O. For example, an application that requires more memory than is available in a single server can access RAM  on other hardware.

Consider that a DDC-Server configured with 1TB of shared memory, 192 cores of AMD processors at 2.8 GHz and 8TB of storage, all connected via an InfiniBand switch and complete with cables, Linux OS and DDC-Pool software lists for $250,000.  A DDC-Server with 256GB of shared memory, 96 cores of AMD Istanbul processors at 2.4 GHz, 4TB of storage with an InfiniBand switch, cables, Linux OS and DDC-Pool software is listed at $99,000.  

Three technologies come together to make this happen. Modern processors support nested page tables, which are a hardware memory management technology in the CPU that allows multiple virtual hosts to have the same view of RAM. 3Leaf leverages nested page tables and their custom ASIC to map memory across multiple computers. The servers are interconnected with a high throughput, low latency (approximately 100 nano seconds) Infiniband switch. Finally, native OS support for ACPI dynamically manages hardware resources.

Representatives from 3Leaf say that a similarly sized system from the likes of HP or IBM might run into the millions, but 3Leaf pricing starts at $99K because they use off the shelf parts for the hardware compared to specialized HPC equipment. At the heart of DDC is a custom designed ASIC and 3Leaf software that is installed on the server hardware called a node. The ASIC runs at Ring -1 that 3Leaf characterizes as below the OS kernel. The firmware resides on a hidden disk partition, and ASIC joins the nodes together and manages remapping the memory address for the OS. When a server is booted up, the ASIC runs after the BiOS initialization but before the OS loads and connects with the other nodes in the rack, and it provides the coherent memory mapping required to share memory among servers. For example, similarly configured servers will have the same memory layout on all the machines, so to avoid conflict, the ASIC maps memory addresses to actual locations either on the local server or on a node. Once running, the OS requires no special drivers and simply treats the resources as if they are local.

3Leaf Vice President Shahin Kahn describes the software as "more like the service processor software that has been typical in large Unix systems. As we go from DDC-Pool to DDC-Share, it becomes more involved and when we get to DDC-Flex it starts resembling a hypervisor. For now, we see it as a service processor kind of software." 3Leaf DDC currently support AMD CPUs, but support for Intel is coming in 2010.

Resources on DDC can be allocated in multiple ways:

 
DDC is the next generation of virtualization and DDC-Flex, if it works as advertised, removes a huge barrier of scaling hardware, allowing administrators to dynamically manage application server performance. Imagine being able to dynamically add and remove CPU cores or RAM as needed. If the DDC systems prove stable and reliable, 3Leaf will turn application deployment and management on its head.