IT Rates IBM's Q1 Labs Top SIEM Performer

Get the full-length IT Pro Ranking report on SIEM

>> See all of our reports <<

Security information and event management, or SIEM, products can help security and IT professionals make sense of the incredible amounts of data generated by security and network devices. They aggregate and correlate events and logs to provide a more complete picture of network activity. Data sources typically include firewalls; switches and routers; intrusion-detection and intrusion-prevention systems; application, database, identity management, and Web servers; and workstations.

While SIEM tools can be useful for security and IT operations, they have a reputation for complexity, partly because of the many data feeds that get connected to SIEM devices, and partly because of the rules and policies that IT has to configure for the products to provide useful information.

InformationWeek asked 322 business technology professionals who use, have used, or have evaluated SIEM products in the past 12 months to rate them on criteria such as performance and cost, as well as feature-specific criteria such as real-time alerting and log management. Our survey listed 17 vendors; of those, eight received a sufficient number of responses to be rated.

The IT pros rated Q1 Labs, which was acquired by IBM in October, tops for overall performance, with a score of 76% out of a possible 100%. Novell is on Q1's heels at 75%, and ArcSight, now owned by Hewlett-Packard, is a close third with 74%. Quest Software, Symantec, and Splunk sit in the middle of the pack with scores in the low 70s. NetIQ and Tripwire are at the bottom with scores of 69% and 68%, respectively.

These overall performance ratings are based 10 general criteria, the most important of which is product reliability, according to our survey. Product performance and flexibility in meeting customer needs round out the top three criteria in importance. That reliability topped the list of general criteria isn't a surprise; SIEM products play a significant role in a company's security operations, and customers need to be assured the product will function well and consistently.

Respondents rated each vendor on these general performance criteria using a five-point scale. On the product reliability criteria, three vendors scored 4.0: ArcSight, Novell, and Q1. Splunk and Symantec were close behind with 3.9 ratings.

Essential Features

In addition to general performance, respondents rated the importance of 11 features found in SIEM products, such as log management and event correlation. Again using a five-point scale, respondents rated real-time analysis for alerts as the most important feature at 4.3, followed by automated log collection from multiple sources at 4.2. Search and root cause analysis and investigation of archived logs were both rated 4.1 for importance.

Our IT pros also rated vendors based on these 11 features. IBM's Q1 Labs ranked highest at 84%. Novell also scored well, with 81%. ArcSight placed third at 77% (see chart below). The features-based ranking showed the largest spread among vendors, a 13-point difference between Q1 Labs and Tripwire, which was rated 71%.

Our report breaks out each vendor's mean average score for various SIEM features (see chart below).

While Q1 Labs earned very high rankings on all the feature criteria, other vendors also demonstrated strengths, particularly on those features rated most important by our respondents. For instance, on real-time analysis, the most important feature, Novell and ArcSight met or exceeded a 4.0 ranking. In search capabilities, Splunk nearly matched Q1 Labs, earning a 4.2 to Q1 Labs' 4.3. Splunk tied Q1 Labs in automated log collection. Novell was the only vendor to score higher than Q1 Labs on any of the feature criteria, earning a 4.2 rating for out-of-the-box compliance reports to Q1 Labs' 4.0.

Why SIEM?

Forty-four percent of respondents say real-time threat detection is a top reason they use SIEM, turning to it to identify potential attacks and policy violations as they happen. This lets security teams respond faster, helping stop attacks at the outset, and reduce damage and recover faster when attacks happen.

A quarter of respondents say "meeting compliance requirements" is the top reason they use SIEM. The Payment Card Industry Data Security Standard, which sets security requirements for handling credit card data, requires companies to review logs daily, including logs from security products such as intrusion-detection systems. SIEM products with strong log management and review capabilities can help companies meet this requirement. Many SIEM products also provide out-of-the-box compliance reporting for regulations and mandates such as HIPAA.

SIEM products must integrate with other security devices, reporting systems, and enterprise management products. Open APIs and SDKs facilitate interoperability. When asked about the tools they integrate with SIEM products, respondents' top responses were network/application configuration management, help/service desk, and performance management.

It's not surprising that configuration management is at the top, given the need for visibility into patch, policy, and compliance information, particularly with regard to vulnerability analysis. Help desk and service desk integration is also sensible, because events and investigations that SIEM products trigger are likely to be logged as tickets within these systems.

Data Deluge

Events and log data from a variety of sources feed SIEM products. Firewalls, application servers, and database servers are the top three sources of event data, respondents say. We were surprised to see intrusion-detection and intrusion-prevention systems listed sixth as these products provide a stream of alarms, notifications, and other data. In fact, SIEM emerged partly as a response to the difficulties that IT and security teams were having in extracting actionable data from reams of IDS and IPS events. One explanation may be that respondents selected "firewalls" as a stand-in for security devices such as unified threat management systems that combine multiple capabilities, including intrusion detection, into a single appliance.

Log management also is now part of many SIEM products. It's not intended for real-time analysis. Instead, it provides a method for forensic analysis of incidents through a normalization of different data sources. Log management also provides a central repository for logs to be stored and archived. While SIEM products may offer some log management capabilities, there is also a variety of products dedicated specifically to log management. In our survey, log management fell somewhere in the middle of the pack in regard to important features. This may indicate that many companies handle log management separately from a SIEM product.

Event and log data that SIEM gathers and searches is likely being stored in a database. Some products use mainstream relational databases, while others have created customized versions of commercial databases. Proprietary databases are another option, often optimized for speed, but possibly with a database schema that isn't open or published. Additionally, vendors may choose nondatabase methods (such as Splunk) that are optimized to speed analysis and correlation. With many customers keeping security data for years, SIEM installations and integrations can even cross over into data warehousing. IT and security pros evaluating these products should examine the underlying database technologies being used to ensure that they're the right fit.

SIEM Challenges

While a SIEM system can be useful, it can also be complex to deploy and operate. Security teams have to set up links between the SIEM system and the devices that feed it events and log data. They also need to build and refine the correlation rules that govern how the SIEM system will respond to the data it gathers and analyzes. And of course, IT must monitor the system and investigate the alerts and notifications.

These difficulties are reflected in our survey. Survey respondents say the top challenge faced with SIEM is managing general complexity (see chart, below). Respondents also cite a lack of integration with other network management tools and building correlation rules. For companies evaluating a product, don't underestimate operational complexity; look for products that offer a user interface that's intuitive and easy to understand and traverse.

Cost also can be a concern with SIEM. Many products are expensive, but the full cost isn't just the hardware and software. You also must account for staff hours and possibly consultant fees for installation and configuration, as well as for the extensive integration required. SIEM products rely on databases for event and log analysis, which means DBA expenses must also be considered, not only for the initial configuration of the product but also ongoing maintenance and tuning.

And of course, IT and security teams will need to be trained to use the product. These factors affect your total SIEM cost. As one respondent commented "Total cost of acquisition and operating is elusive. When you purchase a SIEM solution, the work is just beginning."

Forty-nine percent of respondents who use SIEM say they have no plans to add another vendor or replace the one they're using. Yet when asked what it would take to get them to replace a vendor, the top two factors are substantial savings in capital and operational costs. All things being equal, even those who are hesitant to make a vendor change are willing to consider it if it means a less-expensive product.

And what about the 51% who are considering replacing or adding a vendor? Their top priorities are better performance and operational cost savings. That said, most IT shops aren't rushing to replace incumbent vendors. That's because SIEM products are tightly woven into a larger security management infrastructure and would be difficult to disentangle.

Dean Francis is an enterprise architect at Fusion PPT, a consultancy. Write to us at [email protected].

InformationWeek: July 9, 2012 Issue

Download a free PDF of InformationWeek magazine
(registration required)