Network Visibility Rightsizing

The entire enterprise is becoming tightly data-driven and precision-controlled. Data must be captured from every realm of the business and turned into information that can guide its progress, continually improving efficiency and effectiveness. This data focus is also taking place in networking to achieve high levels of security, ensure compliance and assure network performance and uptime.

When it comes to network data, too much of it is often a major problem. Full capture of network traffic creates a massive processing and storage problem and generates a lot of data that security and network management applications just don’t need. Too much data creates noise and can make it difficult to find valuable insights. It’s the classic needle in the haystack problem—it’s easier to find the needle in a much smaller haystack.

The challenge for security, network performance and management applications is to get the right data from the right traffic. Using metadata rather than entire packets often solves this issue, providing the traffic is from the relevant part or parts of the network the application needs. Getting the right traffic is often no easy task, but let’s concentrate on the process of getting the right level of data—the use of metadata.

Metadata is data that describes other data. It is a summary of the key facts about the packets that is often exactly what an application cares about to perform its function. The other information in the packet would be irrelevant.

Today, there are two primary methods for deriving metadata from packets. The first is a protocol developed by Cisco and made available for their devices in 1996. At the time, it was limited to IPv4, and just a few data fields. Over the years, newer versions expanded it to support more use cases such as IPv6, MPLS and numerous others and has been adopted as an industry standard, supported by many devices and monitoring tool vendors.

The second is the IP Flow Information Export (IPFIX) protocol which was derived from NetFlow V9. It was released by the IETF in 2013 and continues to advance in capabilities. There are clear differences between the two protocols, and it is important to know which is best suited for a particular need.

The most important difference between IPFIX and NetFlow lies in flexibility and interoperability offered by each. Since NetFlow is managed by Cisco, users are limited in monitoring and analyzing their network by their protocol definition.  IPFIX, on the other hand, is designed to avoid these issues, providing more universal support for exporting data to collectors. There are workarounds in NetFlow to increase flexibility—most notably Flexible NetFlow—but it’s important to understand that certain barriers can be avoided when using a flexible protocol, like IPFIX. IPFIX enables custom input of vendor IDs to allow proprietary information to be placed in a flow, exporting all kinds of information without the need for syslog or SNMP collection.

Another key difference with IPFIX is that, unlike NetFlow, the variable-length fields in IPFIX provide access to additional types of information, including messages, HTTP hosts and URLs, which  provide valuable and actionable insight into network behavior. It is also important to consider the application(s) collecting the metadata.  Ultimately, one needs to ensure that the metadata is in a format that their network management tool and metadata collector can interpret with visualizations needed for clarifying how to optimize and protect a network.

It is important to understand what data is needed to perform the particular function. Are full packets necessary to derive the data, or can metadata provide the complete, relevant source? It is also important to understand how the metadata is derived. Does NetFlow fulfill the requirements, or does the application require IPFIX? Traffic delivery systems need to provide the necessary means of getting exactly the right data to the applications it serves.

In either case, metadata is the key to unlocking valuable insights about network behavior. From application performance to security vulnerabilities, user access behavior and root cause analysis, metadata gives the information necessary to keep networks running smoothly.

 Related Network Computing articles:

The Importance of Managing the Entire Lifecycle of Metadata

Auditing and Compliance with Metadata