Web Application Performance Monitors

Hunt and Gather

All the products did a good job monitoring our test bed, which included the default WebSphere Plantstore and Petstore apps (see "How We Tested Web Application Performance Monitors,"). Differences were primarily centered around data collection, for which there were two distinct models. NetIQ's AppManager, CA's Unicenter and HP's OpenView plopped agents down on the monitored systems, while Concord's eHealth, Empirix's OneSight and ProactiveNet gathered statistics using remote protocols and proxies from interfaces, including SNMP, HTTP and perfmon. The benefits of this lighter footprint are countered by agent-based systems' ability to dive deeper into system performance metrics. For example, NetIQ and HP both built automation and diagnostic intelligence into their systems agents. This can be a boon to systems administrators but is less valuable for operational triage and service-delivery reporting.

HP's OpenView especially excelled at collecting detailed data using an army of robotic transactions, flexible deep system agents and, unique to this review, trans-nTier tracking of individual transactions whereby we could assign response times for each tier automatically.

HP and CA also provide network device monitoring and fault consoles built to handle very large networks: CA with its Unicenter Network and Systems Management, and HP with its OpenView Network Node Manager. In fact, these network-monitoring suites are so well-established that the agentless vendors work in tandem with them to overlay and extend your existing network management, by tracking, over longer periods, more about critical interface and network device system performance, such as CPU utilization on a router. This integration has taught them to monitor less intrusively and use what's installed. But the flip side is they need to integrate with and rely on the polling and inventory that network-management frameworks provide.

Important areas to consider when choosing a Web app monitor are administration and implementation; this includes initial installation and ongoing care and feeding. Each product requires planning and setup, much of which can be handled by the vendor's service arm--for a fee, of course. And once you have it in place, the app requires more maintenance than your average supermodel. The many knobs that need to be turned mean that dedicated administration must be part of any TCO calculation.

In our tests, ProactiveNet and Empirix OneSight were easier to get up and running, and NetIQ AppManager had the best administration. At the other end of the care-and-feeding spectrum are HP OpenView and CA's Unicenter, which trailed the pack because of their legacy baggage of multiple products being glued into a whole.Ease of use didn't play heavily into our administration scoring because, make no mistake, you will need dedicated, trained personnel at the helm of whichever suite you choose. But we did want to make sure these hotshots didn't have to spend every coffee break teaching other stakeholders how to navigate the system. In this regard, Empirix OneSight's and ProactiveNet's clean Web interfaces are by far the best. The other products drag along some legacy interfaces that, at best, can confuse newbies and, at worst, actually require IT to jump around to get different views. All the products offer portal-style service reporting of basic information. Concord's eHealth Business Service Console is best at providing multiple service reports for multiple groups, a boon for service providers. HP OpenView offers an integrated operations console with cool parabolic service-topology views, good for documenting service dependencies.

Last and certainly not least is cost. Only the very well-heeled need apply: CA came in at the bargain-basement price of just over $250,000, while HP actually broke the half-million dollar mark. Of course, as with any complex software suite, these prices reflect many ifs, ands and buts; prices as tested are MSRP and can be found in our features chart along with listings of specific components tested. For more on vendor reaction to our pricing scenario, see "Price Skulduggery,".

No product could map our service topologies automatically--the notion that WebSphere server group A and database B are serviced by Web server pool X and all associated dependencies can be easily wrapped into a documenting display is nice, but elusive. HP OpenView did track transactions across our nTier infrastructure, letting us create maps that represent those relationships, but it was still a manual process. Given the coolness of HP's technology, we expect automated service-topology discovery to be the next big thing.After we racked up some hefty phone bills between Syracuse and Green Bay, both Empirix's OneSight and ProactiveNet finished with excellent scores in our report card. Although our Editor's Choice award goes to OneSight, ProactiveNet more than held its own. But OneSight's interface proved a little easier for the uninitiated to understand, and its breadth of integration and monitoring is better than ProactiveNet's. We would recommend any of the top five finishers, however, and though Unicenter lagged in ease of administration, CA shops used to all the buttons and knobs should consider it.

From the get-go, Empirix OneSight just worked. We had the fewest problems getting it up and its agents implemented. It monitored WebSphere on par with rivals. The HTTP PMI (Performance Management Infrastructure) interface extracted performance data from WebSphere and gave us a clear view of the availability and performance of our Java processes.

OneSight's monitoring includes applications or services, including IIS, Exchange, Apache, Oracle, WebLogic and SNMP. We set out to check on these processes and on indicators of our WebSphere applications' health, as well as the app servers' health. Empirix monitored our servers by sending robotic transactions to the different tiers. The place from which transactions are executed can be remote to the server on a proxy agent, and the results are sent back to the management server. Setting up proxy-agent monitoring, a feature OneSight shares with ProactiveNet and Concord eHealth, was made easier by OneSight's display of all available WebSphere PMI metrics.

OneSight did not need to place agents on the systems under test--the products from CA, HP and NetIQ did. We set up a remote proxy agent behind the firewall at our Green Bay labs on a third server to monitor our two WebSphere servers. We then set up HTTP transactions into our Green Bay Plantstore and Petstore apps from Syracuse and set out to monitor our network infrastructure in both lab locations This way, we kept our finger on the pulse of each device and service.OneSight's lack of network device monitoring and fault collection inflated Empirix's price. Empirix made it clear it would not welcome the opportunity to monitor our entire network! In our price scenario, monitoring 2,500 network devices added $300,000 to OneSight's price tag compared with the $300 it costs to integrate into an existing network management application using SNMP traps.

Empirix OneSight 5.0. Empirix, (866) EMPIRIX, (781) 266-3200. www.empirix.com

ProactiveNet has good monitoring and can automatically set values that determine when performance is abnormal so that an alert can be generated. This automated thresholding and an excellent root-cause suggestion engine helped ProactiveNet stand out. Out-of-the-box functionality and low-cost administration are its other hallmarks.

We installed Sun Solaris 9, ran through the ProactiveNet system requirements, installed and implemented it--all in one day. Empirix OneSight was the only other product to be this quick out of the gate. Installation of the administrative console was likewise a simple download off the server. Implementing agents required that we knew which systems were running specific applications, but was again a point-and-click process.

Once we started collecting data, ProactiveNet kicked off an automatic threshold-creation process. It's not unusual for products to include static-threshold suggestions. ProactiveNet instead monitors actual performance and, over time, recognizes aberrant behavior. Static-threshold overrides also are available. We implemented both and quickly found that some of our static-threshold guesses were wrong. Normally we'd reform our thresholds based on the monitoring, but with ProactiveNet it was easier to let the app watch and change thresholds to reflect what's normal for our network.To test the WebSphere applications, we installed a proxy agent behind the firewall in NWC Inc. We downloaded small, self-extracting agent installation files. Like Empirix OneSight, ProactiveNet accessed WebSphere's PMI over HTTP, exposing a large array of possible data-collection points, and ProactiveNet let us choose our own after consulting with our friendly neighborhood developer.

To set up service monitoring, we created end-to-end transactions using included robotic agents. We then created groups to represent critical-path network devices and shared system devices. We also set up groups that used some robotic transactions from each of the WebSphere applications. To test ProactiveNet's role-based access, we defined SLAs (service-level agreements) based on user groups. Worked like a charm.

The GUI is reasonably intuitive. We did get lost a couple of times when setting up groups and adding monitors in the administrative GUI, and the user GUI has an awful lot to look at, but it's not difficult to navigate.

ProactiveNet's price, like Empirix's, was inflated by our requirement to monitor so many network devices. ProactiveNet chose to deliver a volume-price quote. We applaud its forthright disclosure of $50,000 for professional services, something that should be included in your calculations for all the products.

ProactiveNet 6.0. ProactiveNet, (877) 277-6686, (408) 454-4500. www.proactivenet.comConcord defined the performance-management arena and still commands respect for continued innovation. Its eHealth Suite fit easily into our test environment, earning a strong integration score, but the company's focus on end-to-end service availability--something it does well--seems to have left eHealth short when it comes to depth of monitoring the bits and pieces.

As expected, we found that all the products have the monitors available in eHealth, which includes active robotic as well as real-user passive monitoring. One of eHealth's strongest features is the ease with which we could view data. The suite presented us with five basic interfaces: a BSC (business service console), a Web administrative console, a Microsoft Win32/xWindows console, a command-line interface and published PDF reports.

The BSC is the portal-like top level from which service delivery and an overview of services and their components can be accessed most easily and tailored for users. The Web admin interface gave us data on the setup and administration of every monitor, report and grouping, while the Win32 interface, which runs within an X Window console, was the initial point for us to start, stop and license products as well as discover agents and network devices. Reports can be run from this interface, but day-to-day report definition and report viewing are handled more easily in the Web and BSC interfaces, respectively.

For top-level service monitoring, eHealth's views are well-suited to the constituencies they serve. For example, LiveHealth, a real-time display of status and exceptions, will help IT operations see all the components making up a service and how they support the service. To monitor our nTier elements, we had to place active monitors for the separate pieces, dubbed eHealth Availability Agents, on target servers.

Concord does not provide WebSphere-specific monitoring. The company said it feels it's better for ops to monitor systems process, like the WebSphere executable, and leave the detailed PMI stuff to the app server administrator and app dev folks. In light of all the other products' ability to dive deep, we're not convinced.Still, eHealth has a good threshold-management story. It let us set static thresholds by baselining the network over a period of time. A second approach let us apply a standard deviation from what is normal day over day, week over week or hour over hour. As with ProactiveNet, we could define the frequency and consistency for a threshold to be considered violated.

As with all the entries except HP's and CA's, our large network-monitoring requirement drove up eHealth's price.

eHealth Suite 5.6.6. Concord Communications, (800) 851-8725, (508) 460-4646. www.concord.com

AppManager grew on us as we used it. At first, its dense job-management-oriented interface and the difficulty we had getting the agents installed and communicating with the server made our head hurt. But after a while, the product's logical job paradigm won us over--we believe it's a good way to manage and track agent status. Plus, it offers lots of granular systems metrics with flexible ways to apply them.

AppManager is primarily a systems-performance manager with proprietary system agents. Each agent has various capabilities based on the type of device; these capabilities are determined by running predefined knowledge scripts. The number and depth of the scripts included in the base package NetIQ sent us was overwhelming, in a good way. If you're so inclined, you can clone or build your own scripts from scratch, and NetIQ has a large number of user-submitted scripts to work from.Once we had agents installed, we ran a basic systems discovery, then our WebSphere discovery, simply by dragging the knowledge script onto the server or group object. The script was then automatically queued and run from a batch job console. When completed, job status and error data were available within an event console. All consoles are defined within a single Windows or Web interface.

As with its rivals, AppManager uses granular PMI performance metrics for WebSphere monitoring. But the agent integration is very tight--AppManager provided a cohesive view of our system resources.

Network device monitoring is set up like a proxy, similar to Empirix OneSight and ProactiveNet agents, with one big difference: Where most robotic transactions take on the role of client to a target Web server, NetIQ's AppManager network agents can play both client and server using a huge number of predefined simulated transactions. So, for example, two AppManager network agents--one acting as client, the other as server--can pass transactions over the network. Because of the consistency of this performance, which is not influenced by real production traffic, any variation in throughput points directly to the network as the culprit.

AppManager came with a useful selection of predefined views. As agents were populated dynamically on our systems and network devices, they were assigned to the management server inventory that met the requirements of the particular view we wanted. This is a helpful feature--as we added elements, like network devices or WebSphere servers, they were automatically sorted.

It did take some work to get the service view to a bare-bones set of devices that populated just the status of a service--after all, no sense letting stakeholders see the sausage being made. But we did like that the view an end user looked at was the same view that operations could see. That's in contrast to the views provided by Computer Associates and Concord, where separate views had to be created to house portions of information pertinent to the service. With AppManager, these views are unified, just with access controls applied.Access to systems and devices and user- and role-based authentication are defined using the security manager tool. We set up access roles, then defined which groups of devices each role should access. AppManager also let us create different slices of the network, system and application resources. We created views that aligned with our two applications, and created users that had read/write and read-only access to these views.

One thing we would have liked is the ability to set network device access to groups. For example, we had to set SNMP community strings on each network proxy AppManager agent, but would rather have grouped them and set a default for the entire group.

The agent install process can be remotely pushed from the server to Windows servers if the management server and target servers are in the same Active Directory domain. We tried without domains and failed, meaning that, as with the other products, we had to install agents manually on distant servers.

NetIQ also took exception to our network-monitoring pricing requirement, saying AppManager wouldn't likely be used for such a wide network and that its price would therefore be lower.

AppManager Suite. NetIQ Corp, (888) 323-6768, (408) 856-3000. www.netiq.comThis suite provided the most pervasive monitoring among the products we tested because it has the most advanced transactional monitoring. If this had been a review for a soup-to-nuts system that would be adopted by everyone from operations through the database administrator--rather than limited to performance monitoring--HP's suite would have won.

Like CA Unicenter and NetIQ AppManager, Hewlett-Packard's OpenView placed proprietary agents on our WebSphere servers to gather performance information. The OpenView Transaction Analysis and Internet Service components then worked together to track specific transactions from outside the firewall, across the Web server tier, through the application server tier, into the database tier and back out again. This contrasts with all the other products, which monitored all those tiers with discrete robotic transactions and monitors focused on the availability of each defined tier. OpenView did this as well, but added a serial ID to each payload that goes through the tiers. This magic is accomplished using a Class loader call-back hook, which is a byte insertion that wraps the loader with start and stop calls. This interface is supported by both WebSphere and WebLogic application servers.

We installed the WebSphere, SQL and Windows server smart plug-ins. SPIs are specialized agents that monitor specific OSs, services and applications and offer smart event diagnostics with embedded remediation suggestions. Very smart indeed.

Not so smart was the requirement that to finish setting up the transaction analysis agent on our WebSphere machine we had to reboot. Ouch. By default, the agents communicate using Port 135; but because Port 135 was blocked by our host, we needed HP's help to redefine communications to another port (not a simple task). OpenView supports HTTPS and SOAP communications in to WebSphere PMI, but SOAP is not supported on the Windows platform.

We give HP props for including a one-year warranty; you'll pay around $50,000 to upgrade rivals' 90-day warranties to this level. However, HP was the only vendor to tell us that professional services aren't required to implement the product.OpenView. Hewlett-Packard Co., (800) 752-0900. www.hp.com

This suite of products has got the goods once installed and learned, and Computer Associates quoted the lowest price of all the products we tested. For this review, the company sent us Unicenter Management for Web Servers 5.0, Unicenter Management for WebSphere 3.5, Unicenter Network and Systems Management 3.1, Unicenter Service Level Management 3.5 and Unicenter Management Portal 3.1. But count on adding significant professional services and your own resources--this baby needs lots of TLC.

CA has a maze of interfaces and products. Its legacy installed base requires the company to drag along way too many interfaces, but it has added a newer, Java-based Unicenter Explorer. It does not run inside a browser, so the bits will have to be installed, and though the interface is not overly intuitive, it is a definite improvement. This new interface is intended for operators and service-desk administrators, not business users.

Business users will access Unicenter through the Unicenter Management Portal. The portal is, as you'd expect, at the top level of CA's service strategy. It does everything you'd expect from a portal--access control, personalization and, most important, gathering together the far-flung data and applications that are CA Unicenter. CA ships the portal with default dashboards for different management roles. We wanted to track business and IT users of our Plantstore and Petstore services; to do this, we created a couple of portal workspaces and chose representative measurements. For example, we monitored our Web servers using Unicenter Service Level Management and the Web Response Monitor, and WebSphere agents and CA's network and systems management platform.

SLM has agents and collects data by running pings, SNMP queries and HTTP downloads as well as a long list of other methods, on par with the other products we tested. We published performance data through reports viewable from within SLM and the portal. Business-process views (static and dynamic groupings of related systems and network processes that are being monitored) unfortunately aren't integrated with the SLM monitors. Although we could perform service management through correlation of events, doing so was difficult and not nearly as effective as the end-to-end measurements provided by SLM.Agent installation is a manual process. Installing all the distribution files on the remote machine meant mailing the CDs to Green Bay! We begged CA to give us the subdirectory that the agents needed for installation, but the company was unsure what all the dependencies were. Remote installation would have taken way too long across a slow link. The install script did figure out what was needed to install just the agents on the remote servers. But because CA uses separate products for the Web server monitoring agent, the WebSphere monitoring agent and the systems monitoring agent, we had to run separate installations for all three on both our test servers. This compared unfavorably to the more straightforward process that other vendors used, having a base agent that then gains extended functionality.

The install process was rocky, with software components on the server not installing successfully, even though the guided process never indicated failure. We ended up reinstalling from dirt and taking another try at it, which was more successful, but there were still a couple of quirks. For example, our WebSphere servers both showed up in our portal, but we couldn't locate them with the Explorer. Not good for the business owners to know about a service that operations can't find. We eventually found the WebSphere server with CA support's help--it was placed in an "Unplaced Objects" group because of an error in discovery.

Unicenter. Computer Associates International, (888) 423-1000, (631) 342-6000. www.ca.com

Bruce Boardman, executive editor of Network Computing, tests and writes about network management and systems. He has 12 years' experience managing networks and distributed computing for a financial service provider. Write to him at [email protected].

Reliance on or integration with other products is common in the incestuous world of management but created a mess when we tried to pin down pricing. For this review, the impact was twofold: Obviously, the integrated-model vendors (Concord, Empirix, ProactiveNet) have no need to reinvent the goliath network-management-fire lookouts offered by CA and HP. And, they're loath to charge--especially in such a public way as comparative review--for the cost of monitoring the network forest.Some vendors protested the scenario we set forth for pricing purposes, but we feel sticking to a scenario is the only way to do an apples-to-apples comparison. And, by going through the price dance with vendors, we sometimes uncover gotchas that you'll be able to avoid. Not that extracting price is a completely painful process. For example, vendors generally honor our requirement that they provide straight MSRP pricing. This protects their negotiating position with their channels and with you. But every now and then, no matter how hard we strive to make the playing field level, we catch heat from all sides.

This was one of those times. Our pricing conundrum extended even to HP OpenView because of its advanced trans-nTier transaction tracking. To understand why, consider that all the other products track nTier performance by measuring each tier separately and then, when there's a problem, the tier at fault is flagged through a failure registered by the monitor of the failing tier. But because HP can monitor each tier, starting with the Web server, without having to run robotic transactions to each tier, it didn't need as many separate synthetic transactions as did all the other products. Again, our pricing scenario forced all the vendors to price for measurement from 100 locations, each monitoring 10 elements, a significant increase to HP's price.

In this case, we asked for a single out-the-door price. We explained our interest in servicing multiple constituencies to draw together IT and business interests. We prodded vendors to include any professional services that would be needed. And because most of the vendors charge based on the number and type of infrastructure elements monitored, we came up with a list of what we needed monitored. We assumed a distributed network that would benefit from 100 distributed robotic agents and 1,000 passive monitoring agents.

Some of the products are suites, like those from NetIQ, CA, HP and Concord, with optional pricing for various modules, like network monitoring. Empirix and ProactiveNet pretty much bundle everything but are designed and priced to track critical-path and backbone network devices, not entire networks.

For the vendors that don't offer separate network-monitoring options or portals, pricing didn't line up initially because fully monitoring the size network we indicated required network-management products. We believe all the vendors did their best to fully disclose pricing, but your mileage is likely to vary considerably. As one vendor said, "Serious discounting kicks in when you're at this level."Our Scenario:

Network: Heterogeneous routed, switched, load-balanced network incorporating firewalls, VPNs and more than 10,000 ports, 2,500 devices and 7,500 users; 20 Web servers: Apache, Dual Xeon 2.4 GHz, 2 GB of RAM; four WebSphere servers: Sun Fire 6800s; two Database servers: Sun Fire E4900s running Oracle; SAN storage: Sun Storage Edge 6920; Client-side measurement: Microsoft Windows XP; 100 synthetic agents, each running 10 transactions (if applicable); 1,000 passive clients ( if applicable).

We installed all the management stations in our Syracuse University Real-World Labs® and tested IBM WebSphere applications running in our NWC Inc. business applications lab within our Green Bay, Wis., Real-World Labs®. For this test, we used WebSphere's default Plantstore and Petstore applications running on two F5 load-balanced Window 2000 boxes, each running the IBM version of Apache. We installed monitors in Green Bay on separate machines behind the firewall as well as on the WebSphere servers. These agents monitored our network devices, systems, Web servers and application servers. We also installed agents in our Syracuse lab to run robotic transactions and monitor our network infrastructure, tracking the critical network path leading to the applications in Green Bay. To do this, we ran robotic transactions through the WebSphere applications, and performed pings and SNMP gets on the network devices.

On top of this monitoring, we overlaid groupings that represented our two application services. This required that network devices and systems common to both application services be nested along with the monitors of application-specific WebSphere Beans to identify the availability of the application services.

Then we looked for ways to alert operations that one of the WebSphere application servers had failed. Because the servers were in a load-balanced pool, we didn't want the failure of one server to affect the status of the service reports being delivered to the business owners.

R E V I E W

Fax Servers

Sorry,
your browser
is not Java
enabled

Welcome to NETWORK COMPUTING's Interactive Report Card, v2. To launch it, click on the Interactive Report Card ® icon above. The program components take a few moments to load.

Once launched, enter your own product feature weights and click the Recalc button. The Interactive Report Card ® will re-sort (and re-grade!) the products based on the new category weights you entered.

Click here for more information about our Interactive Report Card ®.