Automatically Fix Your Network Vulnerabilities

Security consultancies will tell you that the explosive growth of system vulnerabilities and the risks of not complying with regulatory requirements, such as the Sarbanes-Oxley Act, the Health Insurance Portability and Accountability Act (HIPAA), and the Gramm-Leach-Bliley Act (GLBA), require network architects to purchase vulnerability assessment (VA) consulting.

We say, "Why bother?"

Packaged VA solutions provide an affordable basis for systematic, repeatable methodologies that demonstrate compliance if used correctly (see "VA Deployment Tips" on page 49). The packaged VA solution architectures carry a common theme: They have matured to the point where inexperienced administrators can perform the sorts of security scans and analysis that were once the domain of hardcore security engineers.

What's more, they cost a lot less than VA consulting. We know, because we ran an in-depth TCO analysis of the VA products and services on the market. We priced VA solutions (see "TCO Analysis Details") that will detect and suggest ways to remediate potential application-, transport-, and network-layer holes in a company's security posture. Prices quoted here are list prices; street prices will likely be less. However, the most important items to consider are the internal costs for each solution, so we'll concentrate on those.

We found that while a consultancy's one-time scan of a large DMZ can cost between $250,000 and $350,000, that price would cover a substantial portion of a very large VA deployment, which on average runs $752,000. Third-party consulting services are most appropriate for either one-time or periodic scans and audits of key networks, such as critical servers like those in a DMZ that hold financial or confidential data.VA CHOICES

VA products today come in three variants: software-only, appliances, and services. Software-only based solutions were the first to market and there is no shortage of both proprietary and open-source solutions, with companies such as eEye Digital Security providing commercial software and Nessus being the major open-source alternative.

As the category suggests, software for both the scanning engine and the data repository run on a server of your choosing. The end user--that's you--is often required to install the necessary components, such as the underlying database, and to install and configure the VA console and scanning engine.

If you don't like software-only solutions, consider VA appliances. Foundstone and nCircle are two vendors who represent different approaches to the appliance market. With nCircle, the scanning appliances are hardened Unix systems and the data repository, reporting server, and management console also run on a hardened Unix appliance configured with additional memory and disk storage. The Foundstone FS1000, on the other hand, is a hardened Windows 2000 device with redundant hard disks that can also function as a general server.

VA services use VA appliances, but outsources the management and administration to an external provider, such as Qualys. With the Qualys solution, the VA appliances sit on the customer premises and initiate an SSL connection back to Qualys to receive instructions on what scans to perform. Once the scan data is retrieved by the appliance, the data is sent to the Qualys server farm for storage and analysis. The customer then logs into the Qualys Web site using a unique customer ID, making it possible to run reports or scans, and configure the system.MORE THAN JUST A PRICE TAG

While some costs are equal across products, these architectural differences have a strong impact in other areas, such as capital costs or operational expenses.

An analysis of those architectural aspects shows that based on the price of the vendors' product alone, service-based architectures are the cheapest to implement over a three-year period. Some of this can be attributed to the ability to distribute their costs, such as database support and maintenance, infrastructure, and testing, over a large number of customers.

In the TCO analysis, the most significant cost of Qualys' bid is the yearly licensing cost, which is based on the number of active IP addresses that are being scanned. Because of the Qualys design, there is very little internal expense incurred in implementing and supporting this solution. Based on our model, the supporting scanner cost for a very large-scale deployment is under $10,000 per year. In fact, the majority of the internal costs spent on the Qualys solution is the actual work of scanning and remediation. The appliances are such that installation only takes a few minutes, requiring very little labor.

Next are the appliance-based solutions, such as nCircle and Foundstone. Appliances provide a single, integrated solution carrying the server OS and the underlying hardware. They carry just slightly higher installation and maintenance costs than service-based solution. Still, these appliances take longer to set up and more effort to maintain than services such as Qualys.VA appliance vendors will lead you to believe their products are maintenance free. Don't believe it. These appliances freeze, break, and sometimes become bad network citizens, and thus require human intervention. For the most part, they do not require much attention, but they will fail and they will hiccup. After all, these appliances run their own operating systems and applications. In addition, data backup issues also apply to solutions such as nCircle and Foundstone.

In the event of an error, appliances may be an even bigger headache because they revert to the tools package of their underlying software. A case in point is Foundstone, which is based on Microsoft's SQL Server. While the Foundstone FS1000 product comes with some automated maintenance scripts to simplify installation in the event of an error, architects may find themselves dealing with the complexity of the SQL Server.

At the far end of the operational cost spectrum are software-only solutions. With a software-only solution, the burden of installation and support of the infrastructure falls on the customer. This requires engineers to design and integrate various components, such as the database, the application server, and each scanning engine.

As a result, internal costs are much higher. With eEye, for example, internal costs ranged from $26,700 for a medium-sized installation to $67,500 for the very largest installation. By contrast, all the other architectures came in under $26,000, except for the very largest installations where Foundstone ran $31,800 and Qualys ran $28,050 (see "TCO Analysis Details").

While that's a relatively small amount compared to the overall licensing costs, there are implications on staffing requirements and utilization. In addition, the customer is required to provide the underlying hardware/software for the sensors and management console. Because these systems hold critical vulnerability data, there are extra costs associated with locking down and securing these devices compared to a file or print server. Granted, the actual cost for any specific company will vary depending on cost of labor, skill level of employees, and experience with the product.VALUE MATTERS

Lower costs, however, don't necessarily make for a better solution. Each of the architectures here brings unique value to the market. The service architecture offers a compelling case with ease of deployment and management. Because all of the scanned data is stored at the Qualys data center, there is no effort or cost associated with the maintenance of the underlying database or infrastructure. Qualys is responsible for the optimization of the database and application infrastructure, providing for data fault tolerance (Qualys uses Oracle clustering technology) and troubleshooting database and application issues. Of course, customers must depend on the service provider to fix failures in a timely fashion. And since the connection to the Qualys data center is via SSL, losing Internet connectivity means no running scans, reports on data, or altering the system. While a sustained Internet outage rarely occurs these days, it's a risk that needs to be considered.

Besides connectivity risks, the Qualys solution also separates architects from the inner workings of the solution, so there's less customization that can be done. Qualys does provide an API to access data, but it's not as full-featured as that found in appliance- or software-based solutions. Customers, for example, can't download data based on a custom query. Other solutions, such as Foundstone's, allow customers to see the details of a running scan, whereas with Qualys customers must wait until a scan is completed.

Meanwhile, the term "appliance" means different things to different people. The nCircle appliance has few moving parts and a locked-down operating system, making it cheaper to install and maintain.

By contrast, the Foundstone appliance is essentially a preconfigured Windows 2000 system. Granted, the operating system has been locked down by Foundstone's team of security experts, but it is still a Windows 2000 system susceptible to all of the Windows attacks. While the Foundstone approach shares some of the benefits of an appliance solution, such as ease of management, there is also the potential to incur additional support costs for database and OS issues. As with nCircle, Foundstone's customers can gain access to the underlying data by directly accessing the database. However, nCircle requires that data be exported to another database for custom processing.The most adaptable of the three architectures, however, are the software-based solutions. Architects can dictate, for example, the hardware configuration of the management server. They might choose a highly available and expensive server with dual power supplies, a sophisticated RAID array, and redundant NICs, or an affordable system with just the basics. That flexibility comes with additional internal costs. Someone has to design the solution, buy it, install it, test it, lock it down, and eventually support it.

CANNED SMARTS

VA products have evolved from simple port scanners to sophisticated analysis engines that not only find vulnerabilities but assess the severity of threats.

An example of this growing sophistication is the ability of other security tools to use data gathered by VA products. For instance, an Intrusion Detection System (IDS) can use vulnerability information gathered during scans. VA products define the layout of the network; IDSs, in turn, use information such as the underlying server configurations, OSs, and application sets to better sift through the thousands of alerts seen each day.

With VA data on hand, the IDS can first determine if the vulnerability is present by comparing the VA database against an alert signature. If the target system is at risk, then an alert can be generated. For example, if a MS RPC/DCOM attack is targeted at a Unix system running Apache Web Server, then a lower-level alert can be generated.Another emerging use for VA scanning engines is validating the proper configuration of a system attempting to remotely access a corporate network. If the target system does not meet the defined minimum requirements, such as running all of the critical MS patches, it can be placed into a quarantine network or denied access.

COMMITMENT

While there are different ways of engineering a VA product, the challenges that will be faced in using them are much the same. In order for a VA product to reach its full potential, there has to be commitment to the technology, product, and process. Products within a given architectural area will carry nearly identical ownership implications. There are costs associated with the planning and deployment, such as monitoring scans and working with the vendors on resolving false positives.

Even if architects can find and fix all the vulnerabilities identified by any one of the VA solutions, the work is not done. There are still plenty of vulnerabilities out there, many kept as close secrets by those in the know. There are new low-level vulnerabilities that no one has found. Plus, even if a system has those vulnerabilities fixed, new software can re-install an old vulnerability.

In addition, black hats are moving up the stack; the new targets are application vulnerabilities. Most public Web sites contain internally developed code, and while the application server or Web server may be secure, there are usually flaws within the application itself. Passing a different value in a URL string or POST variable, for example, could allow someone to see another person's data. While this may sound rudimentary, it happens all the time.Vulnerability scanning and detection is not a problem that you can solve once; rather, it's an ongoing process that never ends. New systems are added, old ones are replaced, and existing ones will have software added or modified. The continual scanning and monitoring of results is imperative to an effective VA program. Many VA products now offer the ability to report only changes between scans, such as newly responding ports that indicate a possible compromise and installation of a backdoor program, and newly added systems that may have circumvented the corporate installation process.

Jay Milne is a senior information security consultant with one of the nation's largest health care providers based in Northern California. Milne has deployed and managed VA solutions for both small and large companies. His day-to-day duties include IDS monitoring and response, incident response, and forensics. Send comments to [email protected].

TCO Analysis Details

The TCO analysis was executed using three different implementation scenarios: medium, large, and very large enterprises. Each scenario assumes a different number of active IP addresses scanned, as well as the number of scanners required. The latter will change depending on network architecture. Company politics and business can also affect the number of scanners required. For each scenario, it was assumed that one central database would store all the scanned data. Assumptions were made about the actual cost incurred to perform a task, and it is expressed as the total number of hours per year times an hourly rate. A $75 hourly rate was used, and was based on real-world cost data, averaged from several very large-sized companies. For each product, various internal costs were itemized. Download the full, customizable worksheet at www.networkmagazine.com/tco.

VA Deployment Tips

VA solutions are not plug-and-play. They require planning in order to be successful. Below are tips that will help in a successful implementation.

Socialize the idea, technology, and product before deployment. If you don't, you'll likely face resistance from systems administrators and network engineers to scanning their devices.

Prioritize what to scan and what reports get generated. Identify the critical assets first and report only on the most important vulnerabilities. Input from line-of-business and technology stake holders is essential.Perform both internal and perimeter scans to develop a complete picture of the network. Internal scans are more comprehensive, but perimeter scans will help to determine what information can be extracted by untrusted systems and individuals.

Distribute scanners across key WAN links. A good rule of thumb is to sit appliances at sites with more than 500 active IP devices.

Don't try to fix all issues at once. Start with the most critical, such as vulnerabilities that can give local system access to the remote attacker. While the vendor vulnerability-severity ratings are a good start, security analysts earn their pay by deciding which vulnerabilities to focus on.

Some systems will fail or react poorly during a scan, but avoid the temptation to shut down the scans altogether. The fact that a system reacted poorly to the scan indicates vulnerability.

When faced with a critical situation such as a release of a new exploit, scan specifically for that vulnerability. Don't waste time with general scans.Use the information stored in VA results for risk assessment, asset identification, and management. By scanning the entire network on a regular basis, you can identify the type and quantity of OSs, hardware, and applications. This information is useful when doing a risk assessment of a new vulnerability or exploit.

Establish naming conventions for devices, group names, report names, and so on. Scans make more sense when devices have understandable names.

Scan key critical assets many times per week, depending on your level of concern. Many companies scan DMZ and key infrastructure components three or more times per week.

Risk Assessment: Vulnerability Assessment VA solutions have evolved so that specialized security skills are no longer required to install or operate. There are still false positives, and scans still have an impact on target hosts. In addition, VA products are only now fully utilizing the data stored within the scans themselves to do risk management and threat assessment.

Installing and deploying a VA solution is the easy part. Doing something with the data is the difficult part. Lots of work is involved with VA program management. There can be a lot of internal resistance to scans.

A well deployed and used VA solution can be extremely helpful in securing the network and validating that other security measures, such as endpoint security and patch deployment, are working.

VA solutions can be a critical part of the security infrastructure, but there are plenty of obstacles that must be overcome.