Too Much Information

As information security technology evolves and our infosec tool belts get heavier, most veteran practitioners find themselves longing for smarter--not more--tools. Sure, we have antivirus gateways, in-line content filters, firewalls, scanners, biometric devices, crypto suites and intrusion-detection systems. And we pump thousands--if not millions--of dollars into technology-enabled solutions to our technology-created problems. But try as we might, the Slammers, Blasters and Welchias of cyberspace still tear through our networks like meteors on a collision course. Inundated network defenders may be forgiven for asking, Are the problems getting tougher, or are our "solutions" just not enough, or both?

So you can imagine our excitement when we discovered products designed to take existing technology and make it work smarter. When we first turned our sights on the SIM (security information management) market a year and a half ago (see "netForensics Leads a Weary Fleet,"), we had high hopes.

SIM vendors enticed us with claims that by centralizing logs, using database and visualization tools, and correlating security device and system logs, defenders could decrease workloads and increase efficiency. Who could resist?

Last year's testing, however, told a different story. We found first-generation SIM systems incredibly difficult to deploy, and they delivered only a few pieces of the big picture.

This year, we went into our testing a little less optimistic, a lot wiser and more prepared. When we asked ArcSight, e-Security, GuardedNet, Intellitactics, netForensics, NetIQ, Network Intelligence and OpenService to send products to our labs, NetIQ declined to participate in our tests, and OpenService couldn't get us product in time, so we ended up with six SIM systems to test.Fortunately, we found that this product area has matured a great deal. User interfaces have greatly improved, reporting tools have been refined, data-visualization components are more usable, and the correlation technology has advanced significantly. We barely recognized the netForensics and eSecurity products--they have been completely overhauled--and newcomer ArcSight blasted out of the gates.

SIM technology still has a long way to go, but the progress is encouraging. SIM tools promise to ease the workload and increase the efficiency of overburdened security teams.

Although SIM suites are pricey, the products are evolving rapidly, and we can't imagine running a modern-day SOC (security operations center) without the functionality they provide. In fact, we question the sanity of further IDS spending without SIM correlation (see "NIDS Failure or Market Shift?").

Debates about the future of IDS and firewall technology will rage on, but one thing is certain: Enforcement without an audit component is a risky proposition. You needn't be a statistician to know that firewalls are not blocking all network attacks, and exploits are blowing past basic authentication mechanisms by way of rudimentary OS vulnerabilities. Without further defense tiers and audit mechanisms--and knowledge about how well those mechanisms are operating--security practitioners will continue to fly blind. Monitoring IDS, firewall, authentication, system and application events has become critical to successful security.

But it's pointless to monitor without being able to react once a potential problem is identified. A few hours' delay can mean the difference between a contained piece of hostile code and a global worm outbreak, or a controlled incident versus stolen R&D. Unfortunately, time is money, particularly when it comes to salaried employees and expensive contractors. Incident response, worm containment and investigation efforts are just a few of the tasks that cost companies hard labor dollars.

Security workloads aside, these challenges still make you wonder: Can SIM help, and can it provide the cost-conscious CFO with a measurable ROI? In other words, are these products worth the hefty price?Based on our experience, we believe that, yes, these tools do help, though the savings are heavily dependent on the state of an organization's infosec program. For example, though event monitoring is critical, if an organization isn't committed to monitoring its resources, the cost justification is going to be an uphill battle. In contrast, however, organizations that are determined to run mature security programs understand the need for SIM.

Take log and alert reviewing: The process is tedious with mediocre tools, virtually impossible without them. In our test environment at the Neohapsis labs in Chicago, we had fewer than 10 devices reporting to our SIM systems, and it wasn't uncommon for us to generate more than 20 million events per day. After tuning, the task of reviewing our logs went from impossible to manageable. In larger organizations, the investment in a SIM solution might cut eight hours worth of work down to four for an experienced security analyst, or make the difference between staffing that six-person, 24/7 SOC and arming a few security folks with SIM-connected pagers.

There are tangible benefits in post-event handling as well. For example, trying to gather information after an incident for investigative or litigation purposes can be a time-intensive task, particularly if you have to pull data from multiple domains. Having all of this information centralized and in a system you can query can save a significant amount of work.

Of course, these cost savings are related to how many incident-response and forensics events an organization faces a year, but anyone who has gone through the information-gathering process will immediately understand the benefits.

The bottom line on ROI is that SIM tools can help organizations cut costs on two fronts: proactively reducing risk levels (and in turn, potential downtime) and increasing the efficiency of existing resources. One can argue the ROI of the former, but the cost savings of the latter are irrefutable. Ask five security professionals what ESM (enterprise security management) means to them, and you'll get five different answers. Alert monitoring. Access control. Patch management. Vulnerability assessment. User provisioning. Ask them to define "aggregation and correlation," and you'll likely hear terms like ESM, SEM (security event management), SIM, threat management and security management.We use the term SIM for the aggregation and correlation space because we think it describes the current product functionality more accurately than the other terms, but still, this confusion hints at a bigger issue: What, exactly, do these products offer, and where are they headed?

Right now, SIM systems provide a method of gathering, centralizing, managing and presenting data to the user in a digestible manner. They help with efficiency and visibility into otherwise obscure data sets, but they are a long way from providing big-picture insight. Looking ahead, however, we wonder what other roles these systems may take on. Will SIM evolve to serve as the central security trouble-ticketing center, watching system health, monitoring event data from both operating systems and security devices, or measuring vulnerability and exposure trends?

We suspect the bulk of the command and control (configuration and patch management and change control, for example) will remain vendor-centric. These tasks are just too complicated; most vendors can't even manage their own devices well, much less other vendors' stuff. However, pulling in data from a wide array of critical assets, and providing relevant results from data analysis, is a goal that has eluded security professionals for quite some time. The concept of data mining is not new, but empowering security professionals with analyzed, contextualized, usable information is.

Testing Time: The Early Steps

Using efficiency gains and cost savings as criteria, we began the testing portion of our journey early this summer by constructing an internal SIMnet. Although we learned a great deal from our adventure last spring (see "Connect the Dots,"), unfortunately, this year's journey was almost as rough. Our plan was straightforward: Use existing security devices deployed at multiple locations, and send copies of the data to each of the six SIM systems we were testing.One big mistake we made last year was assuming that we could easily deploy these products without the vendors' professional service teams. We were able to get all the products up and running by ourselves, but we were left with less time than we would have liked for detailed testing and only scratched the surface when it came to feature sets.

We wanted to go deeper this time around, so we had the vendors come in and deploy their products as if we were a customer. This not only saved us time and headaches, it also let us gauge how long the task would take professionals who know the systems inside and out.

The deployment experience was educational on several levels. The professional service teams were much more efficient than we would've been (with the exception of Network Intelligence, because its product is by far the simplest and least resource-intensive of the group). When planning for a SIM deployment, we recommend that organizations use professional services for installation and customization.

And, unlike last year, every participating vendor came to us with detailed questions about device types, versions and logging formats. Although some asked a few of these questions last year, it was clear that the vendors were now inquiring based on a year's worth of deployment headaches--er, challenges--and their approaches were more mature.

Another thing that became clear was how many things can go wrong with a SIM deployment that the SIM vendors can't control--device outages, log-aggregation problems and issues associated with overloading WAN links with event traffic, to name just a few. For example, we deployed a Snort IDS sensor on Syracuse University's perimeter and wound up with more than 400 events per second from a single IDS sensor. This traffic was sent to the Chicago lab and fed (replicated) to all the SIM solutions via a homegrown UDP relayer. However, the Chicago lab's Internet connection is only a T1, and the Syracuse event traffic was chewing up 64 Kbps to 200 Kbps of precious bandwidth throughout the day. We did not anticipate this, and though we solved the problem by tuning the sensor a bit more, you could face similar logjams (see "Critical Questions,").Before our editor finally wrestled this article from us, we overcame other adversities. We encountered device disasters: a catastrophic RAID failure on the Syracuse Snort sensor and the death of an aggregation platform. We had to come up with custom code to replicate syslog data. We watched our WAN bandwidth take a nosedive during worm outages and our event-per-second ratings hop from an

average of 500 eps to 2,600 eps overnight during the worm outbreaks. We had hardware from two vendors fail completely through little more than bad luck, we suspect.

Lest you think we're whining, know this: Outside of needing a custom UDP relayer for our unique "replicate all event traffic to five solutions" requirement, these are challenges that the average enterprise could, and probably will, face. Plan accordingly, and know what risks you're willing to accept when it comes to fault tolerance and the system's effect on your environment. Pilot programs are a must to understand the true operational impact.After initial deployment and tuning, we rolled up our sleeves and integrated these products into our daily lives. We weren't about to chase down the millions of events coming from our university-housed Snort sensor, but we did have our perimeter firewalls and IDSs in Milwaukee and Chicago producing intriguing data, and we found that some SIM features actually made the monitoring portion of security-device management, well, manageable.

For example, by implementing basic filters on event types and severity levels and adding some correlation logic that looked for common attackers based on source address, we greatly reduced events streaming to our consoles. The rule-creation engines in netForensics' and eSecurity's products have been completely redesigned, and we found the new rule-design tools intuitive. ArcSight's correlation tools were also easy to pick up on, and we gravitated to these three solutions more often than the others because it was easier to design rules for them.

We also took advantage of new visualization features. Although purists may mock GUIs and pretty graphs, when sifting through thousands of events and staring at consoles for hours, you'll find visual comfort plays a large role in maintaining sanity. Intellitactics offers the most cutting-edge graphics engine, but we used eSecurity's dynamic graphing tools most frequently. For example, eSecurity'sinterface let us generate eps graphs based on individual devices or groups. When we saw spikes in data from specific devices or certain severity levels, we could easily drill into the data by clicking on the graph.

ArcSight graphs didn't allow for drilling down into events as quickly as eSecurity did, but ArcSight's console could graph just about anything. It was by far the most flexible when it came to charting. Graphing eps rates is by no means a thorough monitoring approach, but we had so many initial problems with lost sensors that being able to glance at a screen and confirm incoming data was a stress reliever.

Other tools reduced data overload as well. For example, we set up asset ratings and mapped event data against those asset ratings to prioritize incident investigation efforts. An IDS alert related to a student's system, an R&D system or even a honeypot will probably be lower in an analyst's investigation queue than, say, an IDS alert related to a production credit-card processing system. We ranked our production systems, or "assets," ahead of our lab systems and test network blocks. By taking into account the target value of an attack or probe, security analysts can prioritize their investigation of thousands of alerts.

We found that the holy grail of correlation and data reduction is taking into account assets, attacks, target operating system knowledge and, ideally, vulnerability information. For example, an IDS alert that indicates a Microsoft-centric attack against a given system is moderately useful at best; an alert stating that Microsoft-centric Attack X was performed on Microsoft-based System Y is far better; and knowing that Attack X was executed against System Y and that System Y was vulnerable to Attack X is an ideal scenario because it all but removes the chance of a false positive. An event correlated and prioritized to this level of detail would be a top priority for any security team.

Unfortunately, most SIM vendors aren't at this level yet. Achieving this degree of correlation intelligence is no easy task. New vulnerabilities are coming out at a rate of 50 to 100 per month, resulting in hundreds of new device-specific IDS signatures and vulnerability checks. Add the need for each correlation solution to support dozens of IDS and vulnerability-assessment (VA) products, and you have a nightmarish maintenance-and-update process for SIM vendors.

Although vulnerability dictionaries, such as CVE (Common Vulnerabilities and Exposures), attempt to create some common methods of correlating products, our own investigation of signature sets and vulnerability checks showed a 60 percent crossover rate in best-case scenarios, below 30 percent in others. Doing it "by hand" is still the only accurate mapping method we know of.

ArcSight appears to be the furthest along on this front; it's created a high-level taxonomy for generic vulnerability-to-signature mappings and has integrated some

device-specific mappings as well, but the company has a way to go before it can eliminate most false positives on the fly. However, by letting organizations import data from popular VA tools like Nessus and Foundscan, ArcSight is off to a great start.

The cleanest approach to this problem we've seen is from Tennable Security's new Lightning product. Lightning supports signature-level correlation between the open-source Snort (IDS) project and the Nessus (VA) project. Unfortunately, when our evaluations started, Lightning supported only Nessus and Snort and was in beta, but we did get to test drive the most recent version and it looks promising. With the click of a button we were able to reduce 100 Snort alerts down to a single event matched to a vulnerable system. Nice, but a far cry from being able to do this across dozens of products.

GuardedNet is working on signature-to-OS checking (due out next quarter), which, though not as granular as signature-to-vulnerability checking, remains a big step in the right direction, and darn cool to boot.

So which SIM suite is top dog? That depends. The best fit for your environment should be detemined by your top product-feature requirements--analyst review vs. monitoring vs. reporting vs. ticketing--and, frankly, your budget.Given our priorities, ArcSight took our Editor's Choice nod for being, simply, a well-rounded solution. However, it was a photo finish among netForensics, ArcSight and GuardedNet, and all the products we tested gave a strong showing. None earned lower than a C. This is in stark contrast to last year's tests, when C+ was the highest score. Our analysis of ArcSight follows. Read about the other products we tested here.

ArcSight, founded in June 2000 and based in Sunnyvale, Calif., is a relative newcomer to the SIM market, though you wouldn't know it by looking at its product--it boasts one of the most mature and flexible consoles, strong visualization tools, easy-to-use correlation functions, customizable dashboards, a healthy set of usable documentation (something often lacking in this product area), and an in-depth reporting engine.

But ArcSight took top honors because it went furthest toward making our lives easier with the least amount of hassle by providing good visual tools, and letting us classify assets and build some basic correlation rules to reduce alert counts.

ArcSight shipped us two units: a Linux-based, quad-processor Intel system that housed the aggregation agents, the Oracle database and the correlation engines, and a Windows-based system for real-time console monitoring. Larger sites may require the separation of the aggregation, correlation and database components, but for our limited deployment, this configuration seemed to suffice.

ArcSight excels in a few critical functions, all of which we put to good use. For starters, ArcSight's visualization features are incredibly handy and met most of our needs. They include a simulated event recorder and play-back mechanism, graphing tools for trending charts, and capabilities for creating real-time motion graphs. We could have gone overboard building real-time charts, but the adage, "A picture is worth a thousand words" has never been truer than when it comes to reviewing obscure data sets.

For general interface usage and navigation, ArcSight's intuitive menu system let us navigate easily between function groups, and a common "look and feel" was evident throughout all areas. Clear labeling of feature areas, such as "dashboard," "administration" and "reporting," reduced the amount of stumbling required to find the right tool for the job. As trivial a point as these labeling methods may sound, intuitiveness has not been SIM vendors' strong point.And this intuitiveness goes beyond the menu system. For example, the rules builder let us pull down functions and object references when building correlation rules.

Although not mandatory, learning the ArcSight rules engine is easier than, say, mastering the nonintuitive rules navigator found in eSecurity's product.

Another time-saving benefit is being able to run utilities, such as DNS lookups, ARIN lookups and traceroutes, simply by right-clicking on an address anywhere in the interface. We found the port-lookup features especially useful--background information on bizarre port combinations was just a right-click away. We suspect that our fondness for the ArcSight console is partially due to the simple fact that it isn't running in a browser. We understand the advantages of Web-based technology, but we find that, usability-wise, browser-based interfaces lag behind their well-designed native OS GUI kin.

We had just a few hiccups with ArcSight. For example, we had some trouble adding new consoles, and at one point our console-access attempts locked out the admin account and we couldn't find a way to reset it. Not a big deal, except that the documentation left us high and dry on the issue so we had to call the company's support line. ArcSight also doesn't have tools that match eSecurity's agent creator, so customers will remain highly dependent on the company for device support.

Finally, there's the price tag: $162,000 (and that's before you start buying hardware!) for four event consoles and support for 100 devices.ArcSight 2.2. ArcSight, (408) 328-5523. www.arcsight.com

Greg Shipley is the CTO for Chicago-based security consultancy Neohapsis. Tom Oele, a security consultant for Neohapsis, is based in its Milwaukee office. Mike Janowski, John McDonald and Patrick Mueller also contributed to this article. Write to Greg at [email protected] or Tom at [email protected].

R E V I E W

SIM

Sorry,
your browser
is not Java
enabled

Welcome to the SECURE ENTERPRISE's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon

above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

NOTE: In the Sept 25th issue of Secure Enterprise, we miscalculated thescores for two products in our security information management review. While the error didn't affect the product rankings, the grades for the NetForensics and Intellitactics products are now higher by half agrade. The grades shown here reflect these changes.