Network Monitoring And Advanced Analytics

Almost all network monitoring products deliver value to network operations in the same way: They collect data from the network, analyze it, and present actionable information to the network manager. In the process of collecting and analyzing network data, network monitoring vendors create metadata. This process is usually proprietary. These vendors make money by being able to produce this metadata quickly and at scale, with maximum insight into network activity.

The data these monitoring tools process varies from vendor to vendor. Some of them process packets. Others analyze network flows or logs. Some crunch device metrics collected via SNMP. Many do several of these at once. The end result is usually a dashboard with widgets, reports, and alarm management systems that help network operations keep their networks up and running.

For many years, network managers were content to interact with their network monitoring vendors in this manner. They might swivel from a packet-monitoring tool to a network flow collector to a device-metric monitoring tool and back again in a single work flow. That process worked fine for them..  However, in this big-data era, some enterprise network managers are starting to have higher aspirations for the data they collect from their networks.

For Enterprise Management Associates’ (EMA) new research study, “Network Management Megatrends 2016: Managing Networks in the Age of the Internet of Things, Hybrid Clouds and Advanced Network Analytics,” we surveyed 150 enterprise network management professionals on this subject. We asked them if they were applying advanced analytics to network data to gain more infrastructure and business insights.

Forty-nine percent said they were doing this today and another 40% said they planned to start within 12 months. These organizations include a wide variety of data types in their advanced analytics. Network flow data (49%) and network security data are the most popular, but not far behind are interpreted packet flows (39%), log data (38%), time-series monitoring data such as SNMP and WMI, (37%), and synthetic network data (32%).

The use cases for these advanced analytics initiatives are broad and have deep impacts on IT operations and the business.  The most popular use case is network security monitoring (38%), which is not surprising given that network security data and network flows (which are often used for network security monitoring) were the top data types used in such initiatives. Network optimization emerged as the next most popular use case at 32%, followed by  business process optimization (27%). When a network team can optimize business processes by doing more a deeper analysis of network data, then you’re really getting somewhere. You are delivering tangible results to the business. You’re not just a utility anymore; you’re a partner.

analysis.jpg

Here’s where EMA has some advice for network managers and their vendors: Open up your metadata to third-party analytics tools. When we asked network managers to list their top challenges to success with these advanced analytics initiatives, one thing became clear: The networking world needs to learn how to share.

While 25% of network managers said their network capacity is strained by the collection and backup of network data for analysis, 30% said their top challenge involves process and political issues in sharing data effectively among relevant stakeholders.  In other words, the IT pros who monitor their networks either don’t know how or don’t want to share their data with an analytics initiative. Furthermore, 24% said they struggle with normalizing heterogeneous network data types, which means they lack the proper database and analysis tools to pull these projects off effectively. They clearly have work to do.

However, many of them will run into a difficult roadblock. The next top challenge to network analytics involves uncooperative vendors. Twenty-four percent  of network managers said their monitoring vendors do not allow them to export data for third-party analysis.  This is a very prominent problem among large enterprises (5,000 or more global employees), 35% of whom said their monitoring vendors won’t cooperate.

These enterprises need their vendors to provide APIs or native integration with analytics tools. Otherwise, network analytics initiatives will have huge blind spots. When you consider the use cases of these initiatives -- network security monitoring, network optimization, and business process optimization -- blind spots are unacceptable. EMA believes the all network monitoring vendors should free their data. Otherwise, they are stifling customer innovation.