ESG Provides Big-Data Primer

Tackling big data challenges is a growing concern for enterprises looking to mine their data stores, according to the Enterprise Strategy Group. ESG defines big data as data sets that exceed the boundaries and sizes of normal processing capabilities, forcing you to take a non-traditional approach. Big-data sets can range from ten to hundreds of terabytes in size.

Managing big data is an issue because the platforms are expensive and require new server and storage purchases, training in new technologies, building up an analytics toolset, and finding people with the expertise in dealing with it. "What's happening is that you've got different vendors saying they have a big data solution without saying what problem they're solving," says Julie Lockner, senior analyst and VP of data management at ESG.

One of the most important factors is meeting the needs of both IT and business departments, says Lockner. "It seems obvious, but a lot of IT organizations are trying to make an enterprise-class decision without collaborating with business," she says, meaning organizations can be over-invested in one technology and not enough in another.

Organizations that are the most successful with big-data initiatives are ones where IT worked with business to brainstorm on what would be possible. "Business isn't going to know the difference between [Apache] Hadoop [a software framework that supports data-intensive distributed applications] and MPP [massively parallel processing]," she says. They just want the answers to their questions faster and smarter, and then those projects will be more successful, she adds.

A recent survey conducted by ESG highlights enterprises' big-data issues. For example, when asked about the importance of enhancing data analytics activities relative to an organization's IT priorities during the next 12 to 18 months, 6% of respondents said it was the most important IT priority, and 45% said it was one of the company's top-five IT priorities.

When the same group of users was surveyed about data analytics challenges their organization had experienced, 47% named data integration complexity, 34% cited the lack of skills necessary to properly manage large data sets and derive value from them, 29% said data set sizes limiting their ability to perform analytics, while 28% said difficulty in completing analytics within a reasonable period of time, ESG says.

Learn "the technologies and procedures for protecting data with 'Strategy: Stop Illicit Data Dumps'" by subscribing to Network Computing Pro Reports (free, registration required).