A MoM with SMARTS

In addition to BMC's Knowledge Module information, we collected SNMP data from Hewlett-Packard Co.'s HP OpenView NNM, IBM's Tivoli NetView, and directly from SNMP-capable network devices. This additional data provided network and system performance and topology information and let us determine how well each MoM integrated with third-party data sources.

We based our grading on five factors: event management, usability, architecture and the pricing for two different scenarios (see the MoMs report card).

MoMs use event management to reduce, focus and make sense of the sea of reports that networked applications create. Event management encompasses simple tasks, such as filtering and "de-duplication" (simply tallying reports of repeated events); moderately complex ones, such as fault suppression (essentially ignoring downstream events); and truly sophisticated tasks that give these events meaning. Normalizing, or simplifying the event description, for example, lets an operator read the description and get a clue as to what has happened.

The most difficult task, meanwhile, is to correlate events to determine the problem's root cause. Of all the products, Smarts InCharge handled correlation best. This product, with its root-cause suggestions, provides a percentage of assuredness for each possibility and helps get to the real cause better than any other MoM we tested.

We considered the products' usability as well. In the simplest terms, MoMs must be intuitive enough to lessen learning difficulty but offer enough flexibility to get out of the way of an experienced operator by offering customizable, predictable navigation and tools.Furthermore, these products must be portable--pushed to customers via Web consoles. In the usability category, we found Managed Objects' Formula's innovative GUI to be especially useful, though this factor didn't make up for some of the product's shortcomings.

The massive number of events created by networked devices and applications requires a robust architecture. Architectural considerations include how distributed and redundant the system is and, of particular importance for a MoM, the ability to integrate into third-party management products. All the products offer distributed processing and redundancy, letting them scale, but both Micromuse Netcool and BMC Patrol Enterprise Manager had distributed engines for filtering event streams close to the event source.

The issue of price is always important, especially with a product class of this magnitude. These products typically set an enterprise back by hundreds of thousands of dollars. That said, any product's ability to reach across the network and application management silos makes the cost reasonable if the application's downtime has enough dollar impact to justify these heavyweights.

We set up two scenarios--one a single site with 5,000 managed entities, and the other a huge organization with more than 100,000 entities contained in 500 branch offices and five regional offices. In calculating total cost of ownership, we included the manufacturer's suggested retail price, maintenance fees, professional services and training expenses (see MoM pricing chart).

Best price? Aprisma's Spectrum xsight for a single site: $66,000 plus $13,200 annual maintenance, with training included. Curiously, Aprisma rang up the highest price in our multisite scenario: $468,000 plus $93,600 annual maintenance and $1,925 per user for three days of training. Unlike the other vendors, Aprisma recommended deploying servers at each of the sites in the second scenario, whereas the competitors say they would locate their servers centrally.Smarts InCharge won our Editor's Choice award, primarily because it handles correlation better than any other product we tested. Aprisma's Spectrum followed closely, thanks to its strong usability and correlation abilities. Aprisma's super-low pricing for our single-site scenario also makes the product worthy of a Best Value award, despite its high quote in the larger setting.

Micromuse Netcool/Omnibus and BMC Patrol Enterprise Manager have the best architecture for focusing lots of events, but neither has the superior event correlation abilities shown by the products from Smarts and Aprisma.

All these products are complex, so plan on reading the manual thrice. And have your wallet ready: You'll need professional services!

Only two products--Smarts InCharge Solutions Suite and Aprisma's Spectrum xsight--perform Layer 2 network discovery. The fact that Smarts stores the discovery results in an object-oriented fashion helps give the product a deeper understanding of the network topology and greatly aids its ability to correlate events accurately.

In fact, this discovery method is a big part of the secret to Smarts' success and garnered the product our Editor's Choice award. For a product to understand the significance of an event in any network, a basic level of topology must be known.Not that InCharge discovered our network perfectly; we had to run the discovery process many times to get the filters right. Furthermore, InCharge's manual configuration process slowed us down. However, the way InCharge handles the process, by breaking down the network into its address ranges and providing many filters for each range, makes the product extremely flexible and accurate. Ultimately, we were impressed with the representation of our BMC Knowledge Module agents, and the correlation between those agents and the SNMP host we had discovered.

Correlation

Codebook correlation, with its unique use of symptom signatures, is Smarts' patented secret sauce. InCharge scans the state of the network, deduces the problems based on the symptoms noted, and gives a percentage of assurance as to the cause.

This technique sounded too good to be true, but when events came pouring in, it worked. Beyond the usual severity color-coding and event de-duplication, we got some clue about what happened to what part of the network and why. When one of our routers' available memory created an event, for example, we could see how often the threshold rose, what devices were affected and the events that ensued. We also found events that indicated high utilization on different ports--an obvious direction to begin looking to fix the problem.

Smarts InCharge lightens the load by simplifying the number of event states a network device or service can have. There are 14 states, to be exact. Among them, hosts can be unresponsive or degraded; applications down, degraded or impacted; and network devices can suffer excessive temperature, backplane overutilization, error rates and instability.Smarts has work to do in classifying enterprise events and traps into one of these few states, but the vendor is actively providing updates. Besides, we found that most of our events were covered right out of the box.

Getting Around

InCharge's object-oriented design and codebook correlation slowed our understanding of the product. The GUI layout isn't intuitive and would have benefited from context-sensitive help. To its credit, we found a complete definition of all the possible statuses that could fit the fields of any event we didn't understand. Once learned, the interface didn't impede us. The system is fully instrumented at the command line, as were all the products we tested.

Although InCharge has a Web console, it's not a downloadable Java applet and lacks some right-click, context-sensitive and drop-down scrolling menus we'd like.

In our pricing scenarios, Smarts InCharge registered on the low side, at $85,000 for our single-site and $169,000 for our multisite scenario (plus 18 percent of the list price for annual maintenance in both cases). Smarts recommends three days of professional services for the first scenario and five days for the second, at $2,500 per day. Training, at $1,000 per day, should require just a couple of days and would benefit from some hands-on experience between sessions.Smarts InCharge Solutions Suite, starts at $35,000. System Management Arts (Smarts), (914) 948-6200. www.smarts.com

Aprisma Management Technologies Spectrum xsight | Micromuse Netcool/OmniBus | BMC Software Patrol Enterprise Manager | Managed Objects Formula and Business Service Analyzer

Aprisma Management Technologies Spectrum xsight

One of the MoM matriarchs, Spectrum xsight has been around since 1991. We've tested four different versions of this product and can honestly say it's gotten better with age.Aprisma, which builds Spectrum, was originally part of Cabletron and is now in the shadow of Enterasys. Aprisma intended to spin out in February, having performed in the black to the tune of $3.4 million, according to SEC filings. The IPO was put on hold at the last moment as the SEC moved in to investigate Enterasys. Aprisma says its business continues to improve and as soon as the SEC investigation lifts, it will launch the IPO.

Spectrum has the best network discovery configuration application, as it allows for discovery by address, range and type. Any of these can be assigned to separate schedules, so we could set differing frequency for discovery of the backbone (an area that changes little) and user subnets (which change frequently and sometimes in bad ways).

The resulting discovery isn't bad either. Like InCharge, Spectrum supports Layer 2 discovery. But it's not a perfect process. During our testing, we discovered connections between devices that were no longer on the network. Even if the error was due to a faulty cache, it was annoying and time-consuming. Also, devices to be managed can be selected manually. Spectrum refers to this process as modeling. Unlike InCharge, Spectrum includes "not" filters, which makes creating subnet and device filters much easier.

The integrated network discovery found all of the hosts, but a separate BMC integration piece was required to integrate the BMC Knowledge Modules. Viewing the KMs also requires additional software on the SpectroGraph, Spectrum's Motif-like console.

Strangely, sometimes the discovery ran rather quickly, and sometimes it seemed to hesitate. It may have been due to the load on the server (the vendor had no explanation), but this process of discovery and modeling can be distributed, which we like as a production control function in systems or network operations.Events

Spectrum's new and vastly improved Web console, Web Operator, is a great place to view events and topology information. Selecting an alarm showed us the views that included the object, such as topology and organization.

We had some good results when looking at events from our Knowledge Module test bed. Downstream suppression has been Spectrum's hallmark for years; it maps ports to attached devices and makes the determination to suppress events based on that connectivity.

In the case of getting a memory event from a Microsoft Windows NT server, we got summary information and could drill down to get specific, accurate probable cause.

Another event, relating to an IBM AIX Sybase server, had less detail but did supply enough information to make a clear determination that Sybase had the problem. In both cases, we found the event's source and determined what devices contained and were adjacent to the problematic equipment. Correlation and root cause were specified in the Alarm Manager client.Spectrum deduces root cause using its own secret sauce--Inductive Modeling--but it isn't as obvious as with InCharge. Although the product has good downstream suppression, de-duplication and Layer 2 connectivity, the BMC agents had to be attached manually to the servers hosting them. This isn't a huge burden, since it's an occasional task, but it's one we didn't have to perform within InCharge.

Usability

Spectrum's client, with its Motif-like look and feel, has always taken some getting used to, which has nothing to do with Motif and everything to do with the huge number of tools Spectrum has added over its 11 years in the business. However, Web Operator is a very useful Java interface. We highly recommend giving this suite to operations and business units.

The Web Operator screen displays alarms with filters, an alarm ticker that scrolls open alarms across the screen, a device browser, reporting and a custom collection view. The custom collector gave us a fast way to make our own groups. With a couple of clicks, we could check the status of the network devices applications that matter.

Spectrum's architecture is also generously appointed with all the network management modules, the BMC integration modules and server redundancy. Aprisma has taken aim at the high cost of network management in the enterprise, putting its money where most offer only mouth. In our single-site pricing scenario, the package sells for just $66,000, plus 20 percent of the list price per year for maintenance. Training is even included. This price is substantially lower than the prices of the others we tested. Although the multisite price far surpassed that of any other product, Aprisma chose to provide each site with a server, rather than provide several centralized units, as the other vendors did. This option provides the additional benefit of leveraging Spectrum's redundancy.Spectrum xsight, $25,000. Aprisma Management Technologies, (603) 334-2100, (877) 437-0291. www.aprisma.com

Micromuse Netcool/OmniBus

Widely adopted, Micromuse's Netcool/Omnibus is known for its speedy event processing, distributed event filters, and an in-memory database that scales very well. It had a midrange price in our single-site scenario and the lowest price in our large-scale scenario; however, it's more difficult to use than most of the competition.

Netcool has always provided very fine control over the way in which events are viewed. A graphical Boolean filter wizard combined with savable sorting tools lets you quickly build event views for every user, hardcore administrator and business owner alike.For our tests, which focused on MoM functionality, Micromuse decided not to include its performance and topology products, so we had no discovery to perform. In this respect, Netcool acted like BMC Software's Patrol Enterprise Manager and Managed Objects' Formula, focusing on underlying management platforms' event streams. There was no correlation based on topology or any polling of network devices.

On the other hand, Netcool has so many options for filtering and sorting events, it's almost an art to administer. Netcool is an event-viewing development environment. Events can be filtered and saved as Boolean logic, making any combination of filtering possible.

Out of the box, Netcool's correlation really amounts to simple event matching. For example, a down event on a particular node is checked periodically for matching incoming clear events, using the node, module, interface and type of event (node down in this example), to clear the new node up event and the existing node down event.

We achieved visual correlation by grouping like devices. By creating an event filter that represents a network device and systems location class, and creating a map or icon representation, we could monitor the status of all services, then double-click to see more specifics.

Nothing about Netcool is very easy. The event list isn't terribly difficult to figure out, but the product's heritage comes through, having Motif, Windows NT and Web-based clients. This complexity, however, is mostly an administration problem; the admin must coordinate and understand the client's capabilities. Netcool's Webtop is really the future for Micromuse, since it makes most of Netcool's functionality available remotely. The Webtop was okay, but had minor failures, such as not bringing up help files consistently.Netcool/Omnibus, starts at $150,000. Micromuse, (415) 538-9090, (800) NETCOOL. www.micromuse.com

Software Patrol Enterprise Manager

Patrol Enterprise Manager has been around since 1989, but even before that, it existed for years as Command Post, from Boole and Babbage. BMC Software updated the interface, adding wizards and Web clients, while retaining a very rich architecture for gathering events. Yet, Patrol Enterprise Manager's heritage does not mean that it has any tighter integration with the BMC Software Knowledge Modules, though BMC has a road map indicating a deliberate move to have all products work within the same Web environment in the future.

BMC Patrol Enterprise Manager doesn't care how many managed nodes, site locations, or managed objects you have in an environment. It discovers the enterprise based on events it receives. This is one reason that Patrol Enterprise Manager is so scalable. Like Managed Objects' Formula and Micromuse's Netcool products, Patrol Enterprise Manager relies on the domains it's managing to provide network inventories. This works when another network management application is firmly rooted. The downside to this is that topology-based correlation depends on an external source.Event management in Patrol Enterprise Manager is really about data collection. Patrol Enterprise Manager has had years to collect events from many different serial devices and mainframe applications.

The Active Alert Display (AAD) is the center of Patrol Enterprise Manager's event display universe. Like Micromuse's Netcool, Patrol Enterprise Manager has filter engines that preprocess events and cut down the number of events allowed into the system. These preprocessing engines are distributable, so event processing can happen close to the source of the events, an approach that has proven to be very scalable.

The AAD in our test bed's BMC Software Knowledge Module did a great job summarizing useful information about the events. In our test of an Windows NT memory threshold violation, we knew from the event line that we had a problem with the NT memory usage on that server. We didn't have to dig for additional clarification.

Unfortunately, Patrol Enterprise Manager doesn't show how one event affects the rest of the network devices and services: There's no integration with an underlying topology. So downstream suppression, though possible, is brittle because the rule engines need to be configured to suppress based on specific events. Get a new event, and you need a new rule. Yech!

BMC Software has moved to a unified Web interface. In the current version, the Unix Motif GUI is still needed to configure and administer the product, but the operations interface works well on the Win 32 client. The Web client is really more of a customer or business unit's current status view. BMC has indicated that more of the operational interface will be rolled onto the Web, but for now you're on your own.During testing, we had problems with our operating system and our database. Although we got help in solving both, we concluded that PEM, while very flexible, has plenty of rough edges that will require careful professional attention. This overhead makes Patrol Enterprise Manager fit best where many sites are generating huge numbers of events that need to be filtered.

BMC Patrol Enterprise Manager's prices for our scenarios were $182,500 and $197,500, plus 20 percent of the list price for annual maintenance. The single-site price far exceeded the competition, but the multisite price fell in the middle of the pack.

Patrol Enterprise Manager, prices available from BMC Software. BMC Software, (713) 918-8800, (800) 291-4262. www.bmc.com

Managed Objects Formula and Business Service Analyzer

Managed Objects' Formula wowed us from the moment we got it installed. From its great looking GUI, which sports a must-see topology display, to its innovative architecture, Formula figured to be a very strong contender. All of this made the bumps we experienced during our testing a bitter disappointment. To be fair, Managed Objects offered new code quickly, but frankly, what the vendor calls a finished, shipping product behaved more like beta.

Rather than perform Layer 2 network discovery, Formula represents the inventory as a direct extension of the devices being managed. It's a very faithful representation because of the object-oriented ORBs (Object Request Brokers) that Managed Objects has created for the domains to be managed. These ORBs on the management server directly gather data from the network management systems at an API level. The data is then relayed to the Formula server and displayed in what appears as the native management console.

The look is so close to native that the topology and groups appeared as if we were using our Tivoli NetView test bed's management software. Formula performed the same trick with HP OpenView Network Node Manager and for our test system's BMC Software Patrol Console. With BMC and OpenView we had bidirectional communications, were able to acknowlege events and configure BMC Software Knowledge Modules. NetView was a one-way view-only access.

Given this close integration, it was no surprise when we saw BMC events populate the Formula console. Correlation is accomplished within Formula by grouping network devices and services. This makes errors show up in the group, indicating that there could be problems. Filtering, what Formula calls "Profiles," is very granular, with filters arranged by class, device or regular expression. Each profile could have a separate retention policy for historical analysis.

Formula doesn't attempt to do the more traditional MoM functions of deduplication and downstream suppression; it relies on the underlying management applications for those functions. We didn't get far collecting events within groups, as we had difficulty getting historical event data to register. Initially it appeared that our BMC agents on AIX may have been problematic, and then we had a problem with the internal database under load.The console runs either in a JVM within a browser or with the Java Web Start client. The console worked fine with the JVM, but we could not configure the Java Web client to work. We also occasionally ended up with more JVMs running than we were actually using, leaving a JVM open unused after closing the last window at the end of a session.

We upgraded Formula a couple of times during our tests. The process was easy, but we found ourselves spending a significant amount of time troubleshooting the installation. At $165,000 for a single site implementation and $217,000 for our multisite scenario, plus 18 percent for annual maintenance, this product is among the highest-priced solutions we tested.

Formula and Business Service Analyzer, $165,000 (Formula Jump-Start Package). Managed Objects, (703) 208-3330, (800) 275-6014. www.managedobjects.com

Bruce Boardman is executive editor of Network Computing, testing and writing about network management and systems. He has 12 years' IT experience managing networks and distributed computing for a financial service provider. Send your comments on this article to Bruce Boardman at [email protected]

To test managers of managers, we used a real application environment: Syracuse University's production registration system. This implementation is a multitiered, Web-enabled PeopleSoft application system with a Web server outside the firewall, and Microsoft Windows NT servers providing secure back-end Java applets that execute native PeopleSoft clients against an IBM AIX PeopleSoft Server that has Oracle and Sybase back-end databases. A dozen servers inside and outside the firewall are managed with BMC Software Knowledge Modules, including ones for Novell NetWare, Windows NT, IBM AIX and BEA Systems Tuxedo.We ran the servers on single boxes in all cases but had multiple separate Web/Java, Win32 and Motif consoles where supported. BMC Patrol Enterprise Manager and Micromuse Netcool ran on a Sun Microsystems Ultra 10 with 512 MB RAM running Solaris 8. The rest ran on dual-processor Intel boxes with 1 GB of RAM, running Windows 2000 server.

R E V I E W

Network Toolbox Suites

Sorry,
your browser
is not Java
enabled

Welcome toNETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® iconabove. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.Click here for more information about our Interactive Report Card ®.