Consolidation's Downsides

I was chatting with a storage vendor the other day who noted that fourth-quarter profits were better than anything his company had seen in the past few years. I asked him what was behind the uptick in spending, and he summarized it in one word: Consolidation.Figure 1:

To hear him tell it, customers found money to spend at the end of the year, and the industry reaped the rewards of a couple of years of evangelizing the consolidation is good” story. Okay, I responded. So, why is it that a lot of smart guys in the trenches are telling me that consolidation is the worst thing that ever happened to them?

I quoted a CTO for a major storage company, who probably would prefer not to be named, since his mother company has just issued a white paper explaining why storage consolidation is a really good idea. According to this fellow: “The economic gain rule just doesn’t apply in storage.” My vendor listener was, at this point, struggling to extricate himself from the discussion.

Server consolidation, I went on, seems to have its place. You have a bunch of older servers sitting around, and you basically (and judiciously) consolidate the applications running on them onto a smaller number of higher-performance boxes with multi-core processors and fatter buses. I can see the wisdom of that, provided you have considered access patterns to the applications and other factors such as heat (especially in blades), impact on disaster recovery plans, etc. Even virtualization in the server space might make sense, since it affords a way to partition and to share resources more efficiently and more manageably.Storage consolidation, however, seems to lack a consistent return on investment any way you look at it. By now, I was talking to myself. The fellow was off telling his good fortune to another willing ear.

Which left me alone to contemplate this mythology around storage consolidation. The value proposition seems to rest on a handful of key tenets:

From these four foundational pillars of consolidation that appear in just about all vendor brochures these days, there is intuitive wisdom for many consumers. It seems obvious that creating a one-size-fits-most platform for data hosting would return a lot of value. In fact, however, the opposite is too often true.

From where I’m sitting, the consolidation pitch we are hearing today from vendors is a rehash of the Fibre-Channel-SAN-everywhere stuff we were hearing three years ago and the ILM verbiage we were getting two years ago. As my friends in Arkansas say, “Same dogs, different set of fleas.”

Consolidating spindles does not yield better management at all. Going to a homogeneous infrastructure (all arrays from one vendor) might do so, by virtue of a common management interface, but even that is by no means certain.

Folks with single-vendor storage networks (all EMC, for example) have discovered to their chagrin that the management utilities supplied with the arrays had gaping holes when it came to managing, not only third-party gear, but also some of the gear within their chosen vendor’s own product lines. So, the idea that fielding fewer arrays will automatically deliver manageability improvements is a lot of bunk.

Safer data?
Next, the idea that consolidation will fix backup woes is also highly questionable. The only way to fix backup is to sort your data and identify just the stuff that needs to be backed up, and to start managing backups with both server-centric reporting tools like Tek-Tools or Bocada and library-centric tools like Stor-Sentry from Hi-Stor. Combined, these products give you all the information you need to identify what is killing your backup jobs so you can address it more readily. Storage consolidation contributes nothing to improved backup and may actually increase the risk to data by putting all eggs in one basket (more on this later).As for the claims about improved security as a result of consolidation, I think the industry might just be pulling our collective leg there, too. Encrypting all your data “in transit and at rest” may seem like a good idea from the standpoint of regulatory compliance, but it also adds significant restore time in the tape world, which has troubles enough already. Three hours per Tbyte of restore becomes 4.3 hours per Tbyte when you add encryption.

In the disk-to-disk world, the problem you run into with encryption is key management, unless you embed keys on encrypt/decrypt appliances that purportedly do “bump in the wire” encryption to data on the move. If you go the security appliance route, you run into the twin problems of having to buy products in multiples to get the number of ports you need to front-end your disk arrays. Then you have to find a way to deal with the heat generated by what are essentially a bunch of over-clocked PCs.

Hewlett-Packard just announced a $52K-per-copy heat exchanger you can fit on your racks of encryption appliances. Have any vendors added that cost to the ROI models they have created to justify consolidated storage? I haven’t seen it.

Oh, and in case you decide to go the encryption appliance route, be sure to remind yourself to buy an extra set of encryption appliances and store them at your recovery center for use when the next hurricane, tsunami, earthquake, or SAN failure chases you out of your primary site. That, by the way, assumes that your recovery will be made at the designated site and not at whatever random site your oversubscribed hot-site vendor decides to use to bring your systems back to life.

Doing encryption in server-hosted software may provide the better answer, by the way, as servers get faster and cycles are plentiful. Such an approach doesn’t require consolidation, but it will require that data be examined to determine which are appropriate targets for encryption services.And if you groaned when I said encryption in software, check out a little startup called Bit Armor in Pittsburgh. It might just change your common wisdom about the old “software-based encryption is too slow” mantra.

The V-word again
A lot of vendors have recently been piggy-backing the advantages of virtualization on their consolidation stories. Thing is, you don’t need to have centralized storage to virtualize storage. Symantec/Veritas Volume Manager will virtualize any LUNs it sees. While I personally don’t much like Volume Manager, a lot of folks use it. Just to be fair, I should say at least as many swear by it as swear at it. In addition to host based virtualization, DataCore Software is doing iSCSI virtualization wherever the iSCSI-attached devices are located across the network. Store-Age and FalconStor and all the rest (except EMC’s Invista product) have either come forward with distributed virtualization or have it on their roadmaps.

EMC is the hold out, of course. They seem to prefer array multi-hop mirroring to copy-during-write approaches that can be enabled through virtualization. I presume this is because they don’t want to gut their cash-cow SRDF product or otherwise sabotage their ability to sell you three DMX boxes, two for local synchronous mirroring and one for asynchronous mirroring at a remote location. All that, when your capacity requirements could be served by just one array.

I expect virtualization to occur at the IP layer, using good old fashioned multicasting. Multicasting can create virtualized storage and support copy-during-write, which I call Smartly EXtended I/O (SEXI ™), because CDW is already someone else’s trademark. Ask the guys at Zetera down in Irvine, Calif., about virtualization via multicast. They are doing some groundbreaking work in this space. When it fully gels, IP virtualization will work in the network and will not fret at all about the physical locations of the storage nodes themselves.

Finally, like the SAN pitch a few years ago, I keep hearing that maintenance costs are sure to drop with storage consolidation. This was a fabrication the first time the vendors said it, a lie the second time they said it, and now a simple absurdity.Let’s assume that Frank, Joe, and Mary no longer need to hop on a plane to service a disk array problem in a branch office in Kalamazoo because all the spindles are in-house and on the climate-controlled raised floor. Surely, that saves money, doesn’t it?

But let’s do a little more math: Disk annual failure rates (AFRs) are set at between 3 and 4 percent by the disk drive industry. That means 3 or 4 disks out of every 100, under minimum (and abysmally unrealistic) workloads, will fail in a given year. It doesn’t matter if the disks are located in the data center or somewhere else. Students of AFR say that failures will occur at a statistically predictable rate. So, if you have a couple of hundred TBs of storage on 70-Gbyte FC drives, your statistical failure rate will run about 10 to 15 drives per week by simple math. This doesn’t change one iota because you have consolidated your storage into a single fabric.

In fact, when you add in the complexity of the FC fabric you have just deployed to facilitate consolidation, and the AFRs of the gear used to build that quasi-network, you can expect to have an even higher number of storage interruptions. In Disaster Recovery Guide 2005, a survey named “SAN failures” as the number three cause of downtime, just behind natural disasters and WAN failures. Servers, by the way, were number four.

Management problem
There are gaping holes in SAN management today – particularly in heterogeneous fabrics, just as there are in distributed storage infrastructure. A lot of factors account for this, but the point is, in the absence of proactive management, consolidation does not contribute meaningfully to maintenance cost reduction and may even increase maintenance expense.

Other problems with the consolidation myth are infinitely more tangible than statistical failure rates. More often than not, you consolidate a bunch of data in one place, and you piss off users. The explanation is simple: Consolidation creates choke points. As you direct all I/O requests to a set number of access ports, you will build queues and bad vibes with the user community. Remember what happened when Victoria’s Secret ran that Superbowl ad without notifying their Web guys? The traffic brought down their servers. Same thing can happen when you consolidate storage, only your users will not be as forgiving as those who are used to the World Wide Wait.

The success of companies like Tacit Networks, Riverbed, Rainfinity (now EMC), and others provides the proof of this argument. With Tacit Networks, you surmount the jam-ups created by consolidation by caching files back out to their end-user workplace. As with the security and heat exchangers noted above, the hidden costs of caching appliances and switches can add up to the point where you find yourself asking the question, “Why did I want to consolidate my storage in the first place?”Bottom line: If we are ever going to realize the benefits of networked storage that have been touted by vendors for the past nine years, two things must happen. For one, we need to stop thinking like data center guys and actually begin putting storage in a network where it will be treated like a peer node instead of a peripheral. FC, by the way, forms a channel fabric, not a real network. In so doing, the time-honored 80/20 rule of LAN design must be observed: 80 percent of storage I/O, like LAN messaging, tends to stay within the sub-network or workgroup that generates it. Most current storage consolidation plans I have witnessed of late violate this fundamental tenet of good network design.

The second thing that needs to happen is open management. Without management, storage infrastructure – whether distributed or consolidated – is an accident waiting to happen.

There needs to be a management framework that can be used by consumers to roll their own storage management solutions. It can’t come from the industry, because there are too many vested interests in making sure that heterogeneous storage isn’t easy. It needs to come from open-source development, where selling more arrays than a competitor isn’t the primary goal of the developers.

With an open management framework that enables any and all management utilities to “plug in” – including the little “Johnny-one-notes” in the storage management software space, whose products do one thing very well – consumers can patch the functionality gaps that exist in the management über-suites in the market. This in turn will enable you to deploy storage in the manner that makes the most sense from an application perspective, instead of spending a lot of money on consolidated one-size-fits-most strategies that end up fitting no one’s needs especially well.

In the final analysis, what needs to be consolidated is management. Not hardware.— Jon William Toigo, Contributing Editor, Byte and Switch

Where have you found real value in data center conslidation? Write Jon William Toigo and let him know.

Organizations mentioned in this article: