SNW And Thoughts On Deduplication

As predicted last week one of the first topics of discussion at Storage Networking World was deduplication. Our first briefing was with Sepaton, who had a panel on the topic. As part of my conversation with Sepaton (for hourly updates on our briefings at the show please go to our blog), we had a broader discussion about the future of deduplication as part of the backup process.

It's interesting to consider that as more companies begin to add deduplication capabilities to their back-up software solutions, the need for a hardware-based target begins to be called into question. I am not predicting the end of target-based deduplication anytime soon, but you can see where as source-side deduplication capabilities are added to software, the need for a hardware-target becomes less relevant.

To be clear, software and source-side deduplication still have issues. It almost always means a software switch, or at least adding deduplication from your current vendor, if they have it. It also is not ideal for all situations, most typically high bandwidth (Fibre/10GBE) and large database environments. There is always the concern about CPU utilization at the client, but that will become less of an issue as the client-side software continues to mature and as the processing power of those clients continues to increase. Let face it, virtualization is built on the premise that we have more processing power than we know what to do with, and I don't think Intel is slowing down.

So what is to become of our target-side deduplication suppliers? I don't think they're going anywhere, either. First, they will likely always have deduplication capabilities, and like any other technology supplier, they will continue to mature and advance their systems. There will always be a data source that does not have deduplication built in. Second, all these source-side deduplication software products are going to need a place to send the data, and while some software vendors are quick to point out that this can be "any" disk, it really can't. It has to be something better. The back-up disk target needs to be handled by relatively high ingest rates from a variety of protocols (CIFS, NFS, Fibre, OST). It needs to offer high reliability and of course scalability.

As deduplication becomes a more commonplace offering from backup software vendors, target-side deduplication products will need to evolve into advanced disk backup storage systems that can continue to deduplicate but also handle all the other expectations of disk based backup.