Root Causes Of Today's Storage Problem

At the start of any pitch for any storage product it seems de-rigeur to sagely state that storage is extremely challenging. Vendors then seem compelled to add a couple of comments about Facebook or Twitter (which are apparently proxies for all data growth everywhere!), a picture of--shock, horror--a coal-fired power station, and ensure that the words "cloud" and "big data" are both mentioned on each of the first five PowerPoint slides ... and--bingo--that's it! Case closed.

Then it's back to the standard sales pitch for the shiny-new-whatever, with plenty of details and specifications, and only rarely will anyone tie their product and its capabilities back to the underlying issues before the pitch is done. We've all seen it countless times. In this article I'll dig a little deeper into the root causes of the very real storage dilemmas. Since I have no product to sell you, I can skip that part of things and--in next month's column--I'll tie some of today's real and available technologies back to the actual challenges. Certainly, the cost, provisioning, and management of storage presents many issues for IT users (which I'll detail here), but there are indeed many reasons to be optimistic (which I'll detail next time).

As one often hears in political discourse, "Something must be done." Invariably, the "something" refers to a significant, known set of problems and a paucity of answers, and it is simply apparent that things cannot go on as they are. In the world of SMB and enterprise storage, the first part of the statement holds true. There are indeed abundant known challenges--data growth, consumerization, clouds, big data, and so on. And often, they come with even bigger expectations. The silver lining to the dark clouds threatening the storage ecosystem is that there are plenty of tools and methods that either exist or are well understood, that can control or remove much of the problem ... if vendors and users are both willing to admit the issues and embrace change before it is too late. The contemporary challenges faced by managers of organizational data and storage are essentially simple, but definitely daunting.

Rising Complexities And Costs

First, there is the constantly increasing need for additional resources to support growing data volumes stemming from natural application growth and new workloads--here, certainly, you can think of the impact of the Web, social media, mobility, and--yes--big data.) Many organizations try to handle this by simply--but expensively--throwing more hardware at the problem, and they often end up with massively underutilized assets into the bargain. Next, operational processes are often not caught up with technology innovations, so as IT service delivery becomes more agile (and this is where the various forms of Cloud enter the equation, for instance), administrators struggle to manage with the same old, inflexible processes.

Budget constraints are almost always an issue, and typically these are more stringent and receive more focus in recent years. In addition, as virtual servers and cloud computing are jointly improving provisioning and providing better service levels, end-users are beginning to expect "instant IT." In the old days (let's say, five years ago!), if a user wanted to launch a new application to support a business process, he or she had an expectation that it would take a while--probably weeks or months--to get through the normal channels. But with fast, easy provisioning made possible by virtualization, IT can spin up a new virtual machine in minutes. However, having the correct, appropriate, available, and affordable storage infrastructure behind that VM and application can be a whole different story. As with any problem across all areas of life, the first two stages (or steps) typically require acknowledging that the problem even exists and then establishing a desire to address it. Storage needs "something to be done" before an unsustainable model--the one largely deployed today--begins to cause real damage to IT and the business.Key Challenges In The Modern Storage World

Data Growth: Data growth is probably the only absolute constant in IT, and arguably, it is the inevitable cause of most operating problems. Data growth never abates; it occurs whether the economy is up or down. One can have a delightful academic debate about whether the demand for capacity will always exceed the supply, and therefore drive the development of new technologies (in terms of both capacity and performance), or whether a declining raw price for storage itself drives latent demands that become economically viable at the new cost level. Frankly, it does not matter. When you're at the top of a roller coaster, regardless of whether you were pulled or pushed up the incline, massive acceleration and g-forces are coming! The whole storage ecosystem is strapped in on such a ride.

The New IT Architectures: Virtualization drives and stresses both storage capacity and performance needs in equal measure--and a poorly designed storage infrastructure can be an anchor on the success of server virtualization and VDI initiatives. At the same time, the use of cloud models is increasing. (ESG's own research pins planned cloud expenditures in 2012 at 7% of all IT expenditure). While the attraction of both virtualization and cloud is beguiling, users all too often find that storage--and its overall management--can exhibit a type of "Ponzi" complexity behind the scenes. Very often, specific storage exists for a specific application, and huge operational issues can arise in making data available to others within or outside an organization. Imagine, for instance, if you had to change laptops to go from Word to PowerPoint; and yet, essentially, that is how storage is often deployed.

Uncertainty: With much of the world's new data produced outside of data centers--unstructured, mobile-produced, rich media being among the "new norms"--the ability to predict what storage is needed within those data centers is a crucial but increasingly difficult task. On the other hand, what is certain, is that end-users have high expectations in terms of speed, availability, and immediacy ... irrespective of application, geography or time zones.

Operational Struggles: A lack of flexibility in, and between, many storage systems simply makes completing the data management jigsaw puzzle even harder. Everything must still be protected and backed up ... whether it's a scale-up or scale-out, distributed or converged, environment. Many users have resorted (perhaps a better choice of words would be "are resigned") to over-provisioning and underutilizing their storage, simply to get the job done, even knowing that such an approach equates to throwing money down the drain!

The Crux Of The Problem

Disk technology is 56 years old this year, and tape is 60. The underlying essence of what we're using for storage hasn't changed. There's an old joke about giving someone directions that ends with, "But if I were you, I wouldn't start from here!" But is that where we are with storage? Are we really just rearranging the deckchairs on the Titanic, or is there room for optimism? The truth is that storage management is indeed complex and tough. And--sometimes unfortunately, sometimes knowingly--getting the job done effectively has for years often trumped getting it done efficiently. So, how and why did we get to where we are?

How We Got Here

Commercial computing took hold when one infrastructure stack executed one specific application for one specific purpose. The original mainframe was essentially a glorified calculator. Centralized computing was predictable and controllable, albeit expensive. And it was manageable: one processor system and one I/O subsystem.

Decentralized (or distributed) computing was developed largely to try to solve the economic (essentially those of Capex at the time) challenges of centralized computing. It yielded low-cost, commodity servers--which we promptly plugged into proprietary, large, expensive, monolithic storage boxes! Servers became cheaper and more interoperable, while storage remained proprietary and expensive. In those old days, the server was the thing that cost all the money. You picked your server by your OS. You picked your OS by your application. Storage was a "peripheral." Today, it is servers that are cheap and interoperable, while storage is still usually expensive, complex, incompatible, and just-plain-difficult. In many respects, it is the last bastion of IT awkwardness: the peripheral tail wagging the purposeful IT dog!

Additionally, two other things have changed lately: The economic downturn has put a laser focus on IT efficiency in general and storage in particular, and the lockstep between new storage technologies and new demands for capacity and performance has been well and truly broken. Without change and innovation, storage costs will escalate to an unacceptable percentage of IT budgets, and most likely do so without providing acceptable service levels either. And so we return to the mantra mentioned at the start that "Something has to change." In layman's terms, what is needed is storage that has more automation, flexibility, application- and business- linkages, resource utilization, and management/tuning ease. Never forgetting of course that there is also one thing we consistently need storage to have less of. Cost!

The good news however, is that there is a surprising number of reasons to be optimistic ... and that will be the [more cheerful!] content for next month's article.

Mark Peters is a Senior Analyst at the Enterprise Strategy Group, a leading independent authority on enterprise storage, analytics, and a range of other business technology interests.

Big data places heavy demands on storage infrastructure. In the new, all-digital Big Storage issue of InformationWeek Government, find out how federal agencies must adapt their architectures and policies to optimize it all. Also, we explain why tape storage continues to survive and thrive.