Deduplication And Solid State Disk, A Perfect Match?

Solid State Disk (SSD) is the medium that storage managers are looking at to address storage performance problems in their environment. Deduplication is what they look at to address out of control capacity requirements. Rarely do you see the two technologies linked, but these two may be a perfect match.

Assuming that you do have a real storage I/O problem, SSD will usually eliminate it. In fact, for many storage problems, it's overkill, but SSD is becoming more cost effective than stringing together a group of mechanical hard drives in a RAID group. What is needed is something in between. In other words, a slightly lower performing SSD that is also less expensive could be an ideal solution for most storage I/O problems. While many SSD vendors are exploring the use of higher density MLC based flash for this purpose, another option is to use deduplication.

There are high performance deduplication options available to vendors that can be implemented with little or no performance loss. While the risk of creating noticeable I/O performance loss on SSD is certainly greater than on mechanical drives, as stated earlier, SSD may have some I/O to spare. For that potential trade-off in performance, you get added capacity. Take for example a 500GB PCIe SSD card: these currently go for less than $15,000. Add deduplication with a modest effectiveness of 5X storage efficiency, and that 500GB card may now be able to hold 2.5TBs. Consider the same combination in a server or desktop virtualization environment, and you may be able to take that utilization rate up to 10X or more. For many environments 5TBs is the entire virtualization infrastructure.

Ironically because of the I/O write issues with Flash SSD, inline deduplication may be the right way to implement deduplication with SSD. While the assumption would be that inline deduplication would add significant latency to primary storage SSD, if used in the right environment it may speed things up. As we discuss in our article on Flash Controllers, one of their biggest roles is managing writes. Inline deduplication, in the right environments, could reduce significantly the number of writes being sent to the devices, reducing the amount of write logic that has to be processed by the flash controller.

There are several products currently shipping that perform deduplication on SSD technology, and with the announcement of several primary storage deduplication API sets, more should be on the way. The combination of significantly increased SSD capacity for a small reduction in performance may be a winner.