Patch Managers Need Perfecting

The agent products we tested didn't pose a substantial burden to user workstations or network servers. The agents also don't rely excessively on Windows services, such as Remote Registry and Server services--an added benefit, particularly for "hardened" server environments. The agents we tested ran as services on our client computers. They are easy to install and do not appear to affect working environments. Agent products also let you target systems more precisely, a feature that becomes more important with granular or small-scale patch deployments. Finally, the agent products can retrieve quite a bit of information beyond basic patch levels, including user names, application lists and available system resources, modestly increasing the amount of control you have over patch distributions.

We found the agentless products less informative than the agent ones, but also less complicated to install, and what we lost in information we gained in simplicity. In organizations where resources are already strapped, this may be important.

The agentless products generally performed patching as well as their agent-based brethren. We successfully discovered all our targets, including our standalone systems, with all the agentless solutions except Gravity Storm's Service Pack Manager (SPM) 2000. SPM relies on WINS (Windows Internet Name Service) queries and had difficulty locating systems outside our Active Directory, and even then it couldn't properly enumerate the AD tree.

Rollback: Why Care?

Ever deploy a critical patch without first performing a full regression test, only to be called in by the helpdesk early in the morning to figure out why, suddenly, no one can log on to the network? If so, you probably wanted to undo the deployment magically. Enter rollback.Of the products we tested, only two, Ecora's and PatchLink's, let us roll back patches. A rollback capability is tops on the wish list for the 600-plus respondents to our reader poll--66 percent consider the feature one of the three most important, compared with 55 percent for Windows server support and 51 percent for Windows desktop support.

If we ignore the poll for just a minute and look at how patches should be processed on receipt, we can put these results in perspective. In a perfect world, administrators would receive their monthly patch updates every second Tuesday from Microsoft. The admins would then burn the new patches to CDs and walk over to their test labs (which would be separate from their production networks, right?) and give the new patches a spin. Next, a squadron of testers, from ace programmers to the befuddled CxO with his or her Blackberry, would beat up the systems in the lab to figure out what happens when they're installed in the production environment. The patches would perform flawlessly in testing, and the whole cycle would be wrapped up just in time for the Friday evening maintenance windows. And pigs will fly any day now.

The reality is that most organizations don't have these computing resources and staff, and patches don't always act the way we'd like. Our reader poll drives this point home: 44 percent of respondents said having the time and staff to test patches is the biggest challenge to patching servers; coordinating resources is second, cited by 17 percent. In the real world, administrators rarely have the wherewithal to perform a full regression test on a simulated production network. The result: If the patch is critical, it will be deployed as soon as possible with a modest amount of testing and a lot of finger-crossing.

One caveat: Although we appreciated Ecora's and PatchLink's rollback functionality, not all patches can be rolled back. Some must have an uninstall flag set, with registry keys or an uninstall script. Without this, it would be difficult to uninstall any patch.

That was ThenMuch has changed since we first reviewed dedicated patch-management products in September 2002 (see "PatchLink Helps Keep Windows Closed,). Major revisions have been released and more are on the way. Timely patching is now considered critical to the survivability of servers. Vendors have listened to their customers.

However, even with this progress, we found ourselves craving more improvements. For example, we intended to eliminate products that didn't have rollback capabilities and couldn't limit bandwidth usage, but these stipulations would have left us with only two contestants! We also found most user interfaces lacking; only the GUIs from BigFix and St. Bernard lived up to our expectations. Furthermore, reporting, for compliance and "dashboard" purposes, is still shoddy overall, and only the products from Ecora, BigFix and PatchLink could patch anything other than Microsoft platforms.

In the end, Ecora's Patch Manager took our Editor's Choice award based on its rollback capability and interface usability. If the folks at BigFix include rollback in upcoming versions, its Enterprise Suite will be right up there (a price cut would be nice, too). PatchLink's newest version, 6.0, looks to improve on a few of the previous version's shortcomings and is due out by the time this article goes to print; the product we tested finished third. No matter which software you select, we recommend running a pilot test before making a choice. And, of course, you'll need to determine which product fits your budget. Note that pricing is list for 1,000 end-user nodes and 20 servers with a one-year license. There's a price disparity among the products (unjustified as far as we're concerned), with BigFix's at $20,349 and SPM at $7,695.Ecora's Patch Manager, despite some quirkiness in the user interface, was our winner by a neck. Patch Manager's comprehensive features let it pull away from the pack: It wasn't the winner in every category, but it consistently scored in the top half of our tests and successfully implemented rollback.

Patch Manager's user interface, though easy to work with, has a few quirks that kept it from taking first in the management category. For instance, the Patches view lists patches only by the Microsoft "Q" article number (Microsoft's reference system to its technical documentation). This made it complicated to find additional information about a patch; we would have liked to see at least the Microsoft security advisory number and a brief description. However, with the Q article number readily available, we could quickly find the hotfix and determine if it had been installed on our systems. We could also access CIAC (Computer Incident and Advisory Capability, from the Department of Energy), Microsoft bulletins, Ecora staff notes and a summary of the original Q article.

The Hosts view provides a tree that you can expand to show all systems or the systems contained within groups you created or imported from Active Directory. The tabs at the top let you switch between any and all Windows components, making it easy to find hotfixes that must be installed.The icons in the user interface indicate whether a patch has been successfully deployed or is needed. If a patch can't be marked as installed or not, it's easy to dig into the reasons why. A few clicks of the mouse led us to comments about missing registry values, dependencies or newer versions of DLLs. We found this feature very useful.

Patch Manager's group-administration functions made it easy for us organize our systems, and its integration with Active Directory was the icing on the cake. Patch Manager automatically added systems to groups based on our OU (organizational unit) structure. We could further customize and add groups by using the system-management tool. This flexible grouping of systems is convenient in smaller organizations, critical in larger ones.

Another notable feature of Patch Manager is its repository management. In particular, the Patch Repository Scheduler feature let us ensure that all the latest patches had been downloaded to our servers for the operating systems and software we selected. Once we selected the operating systems in our environment, we could set the scheduler to download all the appropriate patches. This selection capability is much easier than downloading patches manually because the manual process forces you to sort through a sea of patches, including multiple OSs and language formats. Even if you make it through the sorting challenge, you'll need to read the fine print at the bottom of the screen to ensure you have the proper patch. St. Bernard's UpdateExpert and Gravity Storm's SPM came close to achieving an easy-to-use patch repository, but Patch Manager's automated approach was by far the simplest to use.

Patch Manager's reporting was a mixed bag. Reports for patch deployment and system status were detailed but failed to provide a high-level overview and didn't necessarily behave as expected. For example, sorting by risk displays the results in medium, low, high and then unspecified order. Common sense would place the order as high, medium, low and unspecified. Although the reports are comprehensive and provide plenty of options, you can't design custom reports. If Ecora can beef up its reporting and improve the management GUI, its product will stand out as the clear leader.

Patch Manager 3.0. Ecora Software, (877) 923-2672, (603) 436-1616. www.ecora.comBigFix takes a slightly different approach to patch management than the competition. The differences are most pronounced in its product's user interface and agent model.The GUI is stark compared with the colorful icons employed by the other products we tested. However, we found the interface sufficient for most tasks. It's easy to use and integrates Active Directory's organizational capacities nicely. After a few hours, even the most novice administrator should be able to navigate it easily.

An interesting feature of BigFix Patch Manager is the ability to use retrieved properties for systems. We used this capability to create manageable groups of systems based on their IP addresses. The retrieved-properties capacity let us retrieve nearly any property from a client system and create patch distributions based on those results. It did require some patience to figure out because the scripting used for retrieved properties isn't terribly intuitive, but we think it helps this Patch Manager stand out.

BigFix Patch Manager's deployment strategy has some advanced features that push it near the top. Bandwidth management is handled through what BigFix calls Temporal Distribution and the use of Relay Agents. Temporal Distribution lets Patch Manager execute actions over the course of a given time interval. This can reduce the load on a network during large distributions. The Relay Agent model is interesting, too. Any BigFix Patch Manager client agent can be enabled as a Relay Agent automatically so it can act as an aggregation point to distribute patches. You can then assign specific systems to download from specific agents, providing another point of granular control for large-scale patch distributions. If a distribution fails, the action may be retried any number of times; you can wait a given amount of time between retries or wait until the computer has rebooted.

BigFix Patch Manager also stands out in reporting. With just a few clicks, we were able to use any of the canned reports, generate custom reports or understand the big picture of our patch-management efforts. Unlike Ecora's Patch Manager, BigFix's let us create reports about any specific system or patch. For already overburdened administrators, being able to generate "high-level" reports easily is a real time-saver.

If BigFix offered patch rollback and sold at a more competitive price, it would rank at the top of the class. We're looking forward to seeing what the company does in later revisions.BigFix Patch Manager 4.0. BigFix, (510) 652-6700. www.bigfix.comThe first time we tested PatchLink Update, it easily beat the competition, consistently scoring at or near the top of every category in that review. A year and a half later, the competition has caught up. However, even with the more level playing field, PatchLink Update is still one of the more well-rounded entries, and it's the only product we tested that can patch a dizzying array of non-Microsoft operating systems, including Apple OS X, IBM AIX, Linux and Sun Solaris.

During our tests, PatchLink fared particularly well in its ability to roll back patches and detect missing ones, but it wasn't up to speed with rivals in manageability on account of a sometimes awkward GUI and its lack of a substantial reporting ability. For example, the front page of the GUI provided some basic pie charts, but the rest of the reporting was generally difficult to use. Within the management category, the mandatory deployment feature was easy to use and helped ensure that all new systems would be patched adequately.

We found that the group-administration function lacked the flexibility of those found in the products from BigFix and Ecora. For example, the inability to create groups based on IP address and Active Directory structures made it somewhat awkward to organize systems into useful, logical groups. We were able to create mandatory profiles, which can be used to ensure that all systems within a group have a given set of patches, but we still feel that better system grouping is critical.

PatchLink's reporting capabilities could also be improved considerably. For example, we could sort patches based on whether we had installed them, but the list included all patches--regardless of whether they were applicable to our systems. We consider this more of a patch deployment status indicator than a true reporting tool. And beyond one lonely pie chart on the main page of the administration interface, we were unable to dig further into overall status.

PatchLink Update Server 6.0 was released as we finished our tests, but we were unable to include it in this review because of time constraints. Version 6 includes an improved ability to deploy multiple patches at once, a revamped end-user interface that lets users postpone patch deployments and an enhanced admin interface that allows filtering, among other improvements. If version 6 works as advertised and PatchLink updates its reporting engine, the company might be able to regain its throne.PatchLink Update 5.0. PatchLink Corp., (888) 970-1025, (480) 970-1025. www.patchlink.comWe tested St. Bernard's UpdateExpert using its agentless mode. UpdateExpert is unique among the products reviewed in that it supports both models. We found the agentless implementation to work well in deploying patches and querying systems.

UpdateExpert's incredibly intuitive interface and solid bandwidth-management features help ensure that patch installations won't clog your network. Moreover, UpdateExpert is one of the easiest programs to use when sending multiple patches for one or many systems. However, we found it lacking in other key areas. It can't perform rollback operations; it doesn't let you add systems via a range of IP addresses; and it can't download certain patches, such as MDAC 2.8. For example, instead of automatically downloading the patch, it required us to download the patch manually and then direct UpdateExpert to the proper location.

UpdateExpert provides solid Active Directory integration. The ability to browse the OU structure exactly as it appears in the Microsoft management tools means you won't need to learn an additional interface. This let us, for example, select a particular OU to which we wanted to distribute hotfixes. As organizations move to Active Directory, the structure of the directory can continue to be managed without the need to understand a completely different structure--or attempt to re-create it--and UpdateExpert nailed this part of the equation.

Also notable are the product's grouping and profiling options. UpdateExpert let us create groups that we could customize to meet the needs of our systems. This made it easy to display custom groups and deploy applicable patches to these systems. We found the ability to create profiles excellent conceptually, but we'd have liked to have been able to group by system, for example, including systems that are missing patches. This would result in our being able to see which systems don't have a particular service pack or patch installed, if the service pack and patch have been selected in the profile.

UpdateExpert could use some improvement in its reporting capabilities. Each report is a simple Web page. Although easy enough to read, the reports lack that "big picture" overview management often craves. The validation report provides a comprehensive explanation of why any and all patches appear as invalid or as unable to be validated properly, but may be too technical for most managers. UpdateExpert also can't create custom reports.Finally, the user interface, though it provides excellent status messages and categorization abilities, failed to retain the sort order we selected when switching between machines. This made it awkward to compare two similar systems.

UpdateExpert. St. Bernard Software, (800) 782-3762, (858) 676-2277. www.stbernard.comGravity Storm has been in the patch-management space since the early days. Although it remains an extremely affordable tool, SPM lacks many of the features administrators need in their patch managers--namely, rollback and scheduling capabilities and comprehensive reporting. However, SPM may have a place among extremely budget-conscious organizations.

SPM was one of the easiest products to get up and running. Once it was installed, we started scanning our systems using its intuitive interface. Unfortunately, we soon discovered that our systems needed to be running WINS for Service Pack Manager to be aware of their presence. We see this becoming a sticking point in the future--as more and more companies move to Active Directory for their primary authentication and network schema, WINS is being used less. In our testing, we found that when we added standalone systems that were not running WINS, we were unable to add them to our test environment. Clearly, this is a problem.

Scanning was easy: We simply selected systems we had added and clicked the NetQuery button. SPM then scanned and returned results in relatively short order. The results were displayed in a concise manner, but not always as clearly detailed as rivals'. For example, SPM displayed our Windows 2000 systems as Windows 5.0, Windows XP as Windows 5.1 and Windows 2003 as Windows 5.2. While this was little more than annoying, we found it strange that the problem hadn't been addressed.

Finally, SPM's performance in detecting and distributing patches was significantly below that of the other products under test. In one instance, we were unable to install a patch for MDAC 2.8 because its scanning engine couldn't identify the correct file to check the proper version. The error resulted from a simple typo (SGLSRV32.DLL instead of SQLSRV32.DLL) that was built into the product. This inattention to detail is clearly a problem in SPM, which resulted in some avoidable but critical failures.In the end, if price is the single most important factor in your decision, SPM may be a viable option, but we encourage organizations to be aware of the features they might be missing out on.

Service Pack Manager 2000 6.9.7.0. Gravity Storm Software, (858) 792-0162. www.securitybastion.com

TONY ARENDT is a senior consultant for Chicago-based security consultancy Neohapsis. Write to him at [email protected].To create a test bed that would closely simulate a live production environment, we used two VMware GSX 2.5.1 servers installed on dual 2.4-GHz Xeon processor systems with 2 GB of RAM. This gave us a consistent environment that could be reset for each patch manager in a relatively short time.

On each VMware system, we loaded six client virtual machines--two Windows 2000 Professional, two Windows XP Professional, one Windows 2003 Standard Server and one Windows 2000 Server with IIS loaded. All but one system were members of our Active Directory-- the exception was the Windows 2000 Server. We left it out to simulate a typical standalone system that might exist on a company's network.

Our patch-management servers were dual 550-MHz Pentium III Compaq 1850Rs with 1 GB of RAM. We loaded the servers with Windows 2000 Server and Service Pack 4, plus any prerequisites the products called for.We segmented our network using a Cisco Catalyst 5505 switch with three virtual LANs. One VLAN contained our Active Directory server, our patch-management servers and our Internet connection. The remaining VLANs were dedicated to each of the VMware servers, giving us six clients on each VLAN. We then connected the VLANs using a Shunra Storm STX-100. The Storm let us precisely control bandwidth allocated to each VLAN. We used it to simulate the effects of sending patches over slower network links that might experience latency. However, we found this to be of limited value. Although all the products performed adequately when sending patches through a 512-Kbps network link, they took longer to distribute hotfixes over the slower link, as expected. Lesson: Be aware of your network's capacity, and distribute patches accordingly.

Administrators that have slow links to contend with should consider products with remote repositories or aggregation points, or complete bandwidth controls.

Our tests consisted of deploying Service Packs to the appropriate machines along with a bevy of hotfixes, including the latest cumulative patches for Internet Explorer (MS04-004), MDAC (MS04-003) and Windows Messenger service (MS03-043). Additional patches were installed to test the products' ability to batch together patches, provide adequate warning to end users, and perform scheduling and rollbacks.When we announced this patch-management review in late 2003, we were bombarded by public relations inquiries from a dizzying number of vendors. It soon became clear to us that quite a few folks feel they have skin in the patching game. Although we limited this review to products designed for patch management, our recent review of desktop-management suites made it apparent that much of this functionality is already available or coming in suites from Altiris, LANDesk, Microsoft, Novell and many other vendors. Meanwhile, products from companies like Citadel take a slightly different approach. For example, Citadel's Hercules can import data from conventional vulnerability-assessment tools, then push out patches and configuration changes to address the vulnerabilities.

Regardless of the products' background, we're convinced these products are on a collision course. For example, though many desktop-management suites have not been as focused on the intricacies of patching as the specialized offerings have been, Altiris' Client Management Suite incorporates rollback functionality--which only two of the niche patch managers we tested support! That feature helped Altiris land our desktop-management suite Editor's Choice award.

Given the pros and cons of each product arena--and huge price disparities--organizations would be wise to look at both product types before settling on a path. Bottom line: Don't be surprised to see convergence in 12 to 24 months.Under an increasing amount of pressure to help solve the patching problem, Microsoft last year released SUS, or Software Update Services, a free, bare-bones solution for deploying operating system patches to Microsoft platforms. Although the basic package left a lot to be desired, it did give organizations a cost-effective way of keeping critical systems up-to-date in an automated manner. (We didn't include SUS in our tests because its functionality doesn't measure up to that of the other products we reviewed.)We spoke with the folks in Redmond about the company's plans for SUS and found Microsoft has some ambitious goals. For starters, it plans to release the successor to SUS, WUS (Windows Update Services), in the second half of this year. A major part of the plan is to consolidate the patching process into a single mechanism for all patch types (operating system and application), reduce the number of necessary reboots, enable rollback functionality across the board and ensure that all patches use the same installation process by the end of 2004. These would be improvements, and we hope the company can make good on these promises.

Microsoft is also taking an open stance with third-party patch- management vendors by making software-development kits and APIs available to outside vendors. However, we wonder if third-party Microsoft-only patching packages will still be relevant a year from now.

Microsoft won't enter the cross-platform patching arena anytime soon. But with its move to WUS and improvements to its SMS (Systems Management Services), we feel the company's approach to the problem is comprehensive, at least when it comes to Microsoft platforms. Time will tell if the strategy works.