Rolling Review Wrap-up: Web Application Scanners

Companies making heavy use of Ajax that don't want to expose themselves to attack should hire some protection, whether a skilled in-house Web security tester, a SaaS capable of handling Ajax, or the most expensive option, a consultant. With the exception of IBM's AppScan, automated Web application scanners are simply not yet up to the task of finding security flaws in Ajax code. And it's not like we made it hard on them: The Ajax applications we used in testing were relatively simple. None of the vulnerabilities we expected our scanners to find was advanced or required complex analysis of client-side code. Rather, they were traditional Web application security vulnerabilities, just exposed through an updated Ajax interface. As long as the scanners under test could navigate the application, identifying the vulnerabilities should have been a walk in the park.

Instead, most ended up unable to automatically crawl the applications, requiring a human to surf through the site to teach the scanner where to prod and poke.

As we wrap up our four-month Rolling Review series, we do want to award some partial credit. While only IBM's WatchFire AppScan automatically handled our Ajax applications, Acunetix Web Vulnerability Scanner, Cenzic Hailstorm and Hewlett-Packard WebInspect (post-update) were capable of analyzing and detecting vulnerabilities in the Ajax application, albeit only when we manually walked them through the relevant bits.

Unfortunately, that's just not good enough. Much of the value of a scanner is that it's a repeatable, exhaustive crawler. Requiring a human to replace the automated spider reduces the code coverage, and thus the effectiveness, of the scanner. So while we don't give those products a complete failing grade, they have a ways to go before they can claim to be truly Ajax-capable. Until then, expect to dig into code manually.

Long Strange Trip

When we kicked off this Rolling Review, in the May 14 issue of Network Computing, we knew a few things: That browsers are insecure, and RIAs make us nervous. That Ajax is a wildly popular development environment yet very challenging from a security standpoint. And that few developers are committed to secure coding.What we didn't know was whether an automated Web application scanner could handle Ajax, given its complexity. Normally, this code is written in at least two languages, JavaScript for the client plus whatever is running on the Web server.

Then there were the outside forces: Two of the five products were acquired mid-review, while one vendor sued—or is it counter-sued—another for patent infringement. And it's not just the business side of Web application scanning products that's active. The technology itself is in constant flux.

The beauty of the Rolling Review format is that vendors get continual feedback and are able to address issues on an ongoing basis, and in fact, every product reviewed released a significant upgrade during the course of this review. Most added functionality specifically targeted at Ajax applications. WebInspect, for example, definitely deserves a Most Improved award—SPI Dynamics and then, after its acquisition, Hewlett-Packard, have done a great job addressing a variety of bugs and errors, adding new features, fixing checks, and dramatically improving performance on our sample applications. While the history of errors in the product is somewhat disconcerting, WebInspect has strengths as well. If the bugs can stay worked out, HP's acquisition may pay off.

We'd also like to update our take on the risk metric Cenzic calls a "HARM score" in its Hailstorm scanner. We worried that values could be skewed by particular vulnerabilities, and we still believe that quantifying risk in an arbitrary metric is difficult and customers might be prone to misuse or misunderstand HARM. However, Cenzic addressed the majority of our concerns, informing us of options that can tweak HARM reporting to make it much more flexible than we realized at the time the review was written. Additionally, the prices stated in the original article were incorrect, listing ARC as starting at $26,000, when in fact Hailstorm starts at $26,000, and ARC starts at $52,000.

For those interested in the many issues associated with assigning metrics to application security issues, we recommend Andrew Jaquith's "Security Metrics: Replacing Fear, Uncertainty, and Doubt." The section on application security in Chapter 3 provides an overview of mechanisms used to score applications and the advantages and disadvantages of each.Finally, N-Stalker was able to identify and fix the majority of the bugs identified in its review. While the product suffered from a variety of problems, N-Stalker's response and quick release of a new version to address the bugs does bode well. Current customers should look for version 6.0.0.54, which was released on Sept. 25.

Hired Guns

In this review, we allowed each vendor to submit its scanning engine, but Cenzic, IBM and HP also offer their products in a SaaS model, with varying costs and features. Most of the SaaS approaches involve the companies maintaining and running their tools for the customer, much like Qualys has done with its successful SaaS model in the traditional vulnerability assessment market. Additionally, we invited WhiteHat Security, a pure SaaS vendor, to participate.If ease-of-use is your thing, there can be no simpler option than SaaS. By leaving the complex configuration and adjustment processes of scanning to experts at each company, you know you'll be getting the best scan each product is capable of, without having to maintain staff on your own. But is it the smartest line of defense?

We decided to break down the rough costs of three approaches: SaaS, a skilled Web security tester without a commercial tool, and the tester armed with the products reviewed. Now, some vendors offer licensing models that are more attractive with fewer applications to be scanned, while others are based on IP addresses, which can be attractive when many applications are hosted on the same server, so costs depend on your architecture. For our purposes, we assumed five applications that are constantly being upgraded and debugged. We want the five apps scanned once per month over a two-year period so vulnerabilities can be fixed as they're introduced. See our TCO chart //where// to compare what the methods under consideration will cost during that time period.

Bottom line, there are a whole lot of ways to skin this cat. Note that the WhiteHat Sentinel, AppScan OnDemand Comprehensive and Cenzic Click to Secure services fall somewhere in between custom manual assessments which would be priced well above the approaches discussed in the chart, and the totally automated SaaS service WebInspect. A hybrid approach is required to be able to detect vulnerabilities like the Web-mail XSS, discussed later.Besides nabbing all the vulnerabilities discovered by the scanning products, WhiteHat's Sentinel identified e-mail-based XSS vulnerabilities in our sample Web mail application through its combination of manual testing and automated scanning. WhiteHat navigated all our sample Ajax applications without any trouble.

Based on our testing, if you want automated scans of Ajax applications, your best options are Sentinel and AppScan.

Business Unusual

Cenzic sent ripples throughout the Web application scanning community when it sued SPI Dynamics for patent infringement on July 27.

Litigation is normally a last resort in patent disputes. The usual scenario is either a cross-licensing agreement, when both parties have patents the other might infringe, or a licensing agreement. While the lawyers still get their cut, the courts usually stay out of the picture. Indeed, competitors SPI Dynamics and WatchFire cross-license patent portfolios related to their application scanning technologies, and somewhat ironically, HP and IBM, which purchased those two companies, respectively, also have cross-licensing agreements in place.While neither Cenzic nor SPI Dynamics CEO Brian Cohen would discuss details of the suit, Cohen did say that prior to the lawsuit, there were no negotiations under way to establish cross-licensing, or other options to settle claims. When asked for details on the patents that SPI Dynamics holds, he suggested that it would be "difficult for any organization to provide a comprehensive Web application security assessment product that did not employ some of the methods described in our patents."

Whether the end result will be another cross-license or some other outcome is anyone's guess. Further muddying the waters is an odd sense of reverse deja-vu: Last year, the litigation was going in the opposite direction, when SPI Dynamics quietly sued Cenzic. Whether this is simply a case of you sue me, I sue you isn't clear, but one thing is certain: These patent disputes are a zero-sum game for the industry, doing nothing but wasting time and distracting vendors from making a better product for customers.

Cat Out of the Bag

Now that testing is over, we can disclose a few interesting XSS (Cross-site scripting vulnerabilities) that occurred in our sample Ajax applications. All vulnerabilities were in their natural state, exactly as they were written by developers who either made mistakes, or simply didn't know any better. The applications themselves ran the gamut. We started with a piece of software meant to track personnel information, an HR course scheduling and tracking system, and an application used to maintain biological specimen information. While the HR course scheduling application was the only original Ajax application, an open-source Web-mail application was added to the mix. Development languages included PHP, C# and J2EE.

One developer was aware of common Web application security issues and went out of his way to make sure he sanitized user input. Anytime SQL queries were made, user input was carefully filtered and sequestered so it couldn't cause unwanted behavior. Unfortunately, the developer made one small, yet potentially fatal, error: In many of the pages of the application, the current URL was used as the "action" parameter of a form element. Of course, if the current URL contained a double quote, subsequent code would be able to break out of the existing HTML and generate an XSS:

Note the leftover "> at the end from the original

element. This is the type of flaw that an application scanner should be expected to spot. Indeed, once HP and N-Stalker fixed bugs related to this detection, all products were able to find this vulnerability.

Educating developers on security issues is one of the biggest challenges facing the application security industry right now, but unfortunately, even when the developer has been well-informed, it's still all too easy to make mistakes. This is one reason it's so critical to incorporate multiple defenses—developer education, security code reviews, black-box testing—into application development.

A second style of XSS highlights the difficulties in completely automating scanning. The open-source Web-mail project we added, to provide further Ajax functionality for the scanners to assess, suffered from multiple XSS vulnerabilities. We're keeping the application anonymous because not all vulnerabilities have been resolved in released code at this time, however, XSS in a Web-mail product is especially interesting for a few reasons.

First, most Web-mail products attempt to display "safe" HTML formatting so that users who receive HTML e-mail can still view markup on the messages—font colors, say, or images. Unfortunately this is a nearly impossible task to do safely because there are so many vectors and ways to sneak JavaScript into regular HTML. And that indeed turned out to be the case: The Web-mail application was vulnerable to multiple methods of slipping JavaScript into an inbound message. One actually involved turning the security of the application against itself. Because the code tried to sanitize certain contents of inbound messages, we could split a dangerous tag with a tag that would be sanitized. In this way, once the filter to "clean" the code has stripped out dangerous tags, you're actually left with a more dangerous tag.

The < embed > and < /embed > tags were removed, leaving behind dangerous JavaScript that executed in the context of the Web-mail application.Another reason XSS in a Web-mail application is so interesting is that it's relatively simple to turn a vulnerability like this into a self-replicating e-mail worm that requires little user interaction. An employee simply views the e-mail, and the Web-mail session could become infected with a worm that sends itself out to everyone in his address book and steals copies of all his e-mail. This is the type of scenario that caused the security industry to finally take seriously those previously marginalized XSS vulnerabilities.

Sadly, this Web-mail vulnerability is incredibly tricky for an automated tool to find. Not only would the scanner either have to learn how to send mail to the system, or use Web-mail to send messages to itself and then analyze them, it also has to be adaptable and realize that the filtering system can be used against itself.

This type of vulnerability is where a product with a powerful extension capability, such as Hailstorm's raw JavaScript engine or AppScan's Pyscan interface, might be useful—we could leveraging the product's scanning capability and use the extensibility to force the scanner to submit/scan e-mail through the Web-mail to identify vulnerabilities.

So what's the answer? The proper fix to such a vulnerability is to repeatedly apply the filter until the last two applications produce no more dangerous code. This solution can be susceptible to a denial-of-service attack, however, so ensure there's a way for the code to recognize it's being abused, stop deep recursive scanning and simply halt viewing of the e-mail with an error.Scared Yet?

So what should you do if you suspect your code is at risk? Sometimes, there's no substitute for an equally devious human when trying to thwart attackers. But do you do penetration testing? Vulnerability assessment? Application audit? Fact is, these terms are often used interchangeably. That misuse does a disservice to those not intimately familiar with the actions behind the names.

Mike Rothman, author of the "Pragmatic CSO," defines them thus: "Assessments give you an idea about all the potential holes. Pen tests prove whether the holes are in fact actionable."

That's as succinct a definition as you're like to get, but there are deeper differences. First, a penetration test does not set out with the goal of enumerating all risks or vulnerabilities—the objective is usually simply to gain unauthorized access. Assessments, on the other hand, may or may not prove that holes can indeed be exploited, but the goal is to locate as many potential issues as possible to allow an organization to prioritize among all possible avenues to remediate them.

Many audits will fall on a continuum, with elements from both penetration testing and risk assessments. Before you purchase a Web vulnerability scanner, consider what your goal is. Do you need a tool that will help a security engineer demonstrate that flaws are exploitable, or do you just want the broadest possible scan of all potential issues presented in a manageable report?There is the temptation to deride penetration testing and the tools that enable it as unnecessary. Indeed, this opinion is common among security pros. After all, once a vulnerability is highlighted, do you really need it proved definitively? Shouldn't developers fix all potential vulnerabilities?

In reality, the group doing the assessment often isn't the same group maintaining or developing the application. When informing developers of the seriousness of various flaws, nothing drives the point home quite like displaying the entire contents of a database extracted through SQL injection, or showing the developers' very own credentials stolen through XSS flaws.

Additionally, the value provided by a penetration test can be uniquely useful compared with assessments. Say you want to know exactly how easy or hard it would be for a relatively skilled, motivated individual to exploit your application. Do you have any idea of that after a risk assessment? Maybe, but just as likely you don't. After a penetration test, it's crystal clear because, presumably, the actual exploitable holes that used to exist don't anymore.Security is, fundamentally, about managing risk. If your goal is to keep your application secure from script-kiddies tossing low-level attacks for a relatively short period of time, any moderately skilled auditor with a tool and a bit of knowledge of how to exploit common vulnerabilities can give you an estimate of how much hacking your application could endure.

If, on the other hand, the site front-ends sensitive data and is designed to withstand highly skilled attackers for great lengths of time, you are of course using the most stringent secure development lifecycle, multiple code audits and risk assessments. But could it hurt to verify the assumption of safety by hiring one or more highly skilled testers to do their worst?