Security Appliances

The Office

We put on our small-business hats and asked, "What functionality is important to protect our small office with a minimum of IT support pain?" Although we did significant load testing on the participating devices, we knew that for smaller offices, network load is not as important as ease of configuration and a comprehensive feature set. We kept this in mind while developing our test plan. First, we outfitted our small-office network with one WAN interface and one public IP address. Internally, our small office had a local network with private IP addresses and a DMZ for Web, e-mail and FTP servers. Our LAN users were configured using DHCP, and the security devices provided NAT (Network Address Translation) for both DMZ and LAN traffic. The devices also were responsible for port forwarding of traffic between the WAN and the DMZ servers.

The Features

Although not even the best all-in-one device will relieve all your security woes--you'll still have patching to do and need desktop firewalls--we were happy with the devices overall. (For a checklist of other bases to cover, see "Appliances Aren't a Cure-All,".)

The devices sat between us and the outside world, so they had to have firewall features, including traffic filtering, NAT and port forwarding. Stateful firewall features for preventing DoS (denial-of-service) attacks, like syn and smurf, are even better. In addition, some small organizations may want these devices to act as a DHCP and DNS server. All the products provided the basic firewall capabilities we were looking for, but the Fortinet and SonicWall devices can't act as DNS servers, and the Symantec box doesn't provide DHCP functionality.Assuming you've configured your firewall correctly and your server and desktop operating systems are up-to-date, the biggest threat to your organization is traffic brought in from the outside by your users. Therefore, we looked at the devices' in-stream antivirus functionality--meaning the device scans incoming e-mail (and possibly FTP and Web traffic) for viruses. This scanning is similar to what your desktop antivirus software does. If a virus is found, the device can be configured to drop, drop and warn the sender, forward with a warning, or quarantine the e-mail. Only SonicWall's device lacks on-device virus scanning; it takes a desktop-centric approach instead.

Fortinet has a strong antivirus solution, and its device was unique in supporting virus scanning on all the protocols we listed. Neither the Astaro nor the ServGate devices could scan HTTP traffic, and all the vendors require a yearly paid subscription to keep the virus signatures up-to-date.

Another threat is attacks on your servers from traffic allowed through your firewall. That's where intrusion detection/prevention comes in. We grouped detecting DoS attacks, intrusion prevention and intrusion detection under one item. Security industry pundits may disagree with this, but these tasks all relate to monitoring to determine when someone is trying to do something to your systems. For a security device to perform intrusion detection, it must use deep-packet inspection and understand the application, be it a Web, e-mail or FTP server. Vendors have been slow in coming out with this functionality, and only two of the devices tested, those from Symantec and Fortinet, sport what is conventionally known as IDS. Symantec goes further than Fortinet in its IDS implementation: Its device also provides algorithms that actively monitor network traffic for anomalies. The other vendors' devices have limited attack prevention--against DoS attacks, for example--but they don't perform deep-packet inspection.

Content filtering covers how well a device lets you block undesirable sites. We gave each security device a URL blacklist--sites our imaginary users weren't allowed to access. All the vendors in this review provided access to paid subscription services maintaining URL blacklists by group, such as porn or gambling, though we could update the blacklists manually if we wished.

Symantec downloads a blacklist directly into the device; for the others, the list is supported on an external server and only recent lookups are cached on the device. In addition to blacklists, the Fortinet and Symantec devices let us perform content filtering based on file type--.exe or .bat, for example--while all except the Symantec box also filtered traffic based on manually entered keywords or phrases.Finally, we evaluated the devices' VPN implementations. Keeping with our small-office scenario, we were primarily interested in setting up a limited number of VPN tunnels to a corporate office or other remote offices. Therefore, we weren't really concerned with how many tunnels could be established, but rather with the effects of traffic from a small number of tunnels. Our testing involved looking at the throughput and latency for the VPN traffic as well as its effect on normal, unencrypted traffic. All the devices performed well in our basic VPN testing, though Symantec's had the strongest high-end VPN numbers, with ServGate's a close second.For organizations with limited IT resources, a device's management interface can be a deal breaker. If you can't configure a feature correctly or determine if it's working properly without spending days studying the manual, you may have a false sense of security. Thus, we looked critically at each device's management interface. We found that, though security is a complicated issue, configuration doesn't have to be difficult. Of course, there was room for improvement--for example, a message box telling us, "Turn on a proxy or relay agent" was not rated as helpful as a checkbox saying, "Turn on antivirus for SMTP traffic."

We first looked at the ease of configuring the general device parameters and the firewall rules and policies. Then we moved on to configuring advanced features, including antivirus, content filtering, and intrusion detection and prevention. The result: SonicWall is a role model for how to implement a user interface. Its interface has wizards to help with basic device configuration and firewall rule sets, and configuring the advanced features was very intuitive. We had more difficulty configuring the advanced features on the Symantec and Astaro devices, and an even harder time telling if the features were working correctly.

We also looked at the diagnostic tools available from the devices' management interfaces. These tools, which include trace route, ping and DNS lookups, are useful in debugging implementation problems. Other management features we considered included status pages, device-status determination, and logging and notification capabilities. Astaro's System and Network status pages, which display graphs of current device activity, are a standout.

Attack Pack

Of the five devices we tested, two use proprietary hardware and operating systems; three are Linux-based. Although the proprietary devices, from Fortinet and SonicWall, lagged in overall power and features, they were an order of magnitude easier to use and configure. The three Linux-based devices had better functionality and higher throughput numbers but took a little more time, skill and patience to set up.So which device is right for you? For small businesses with limited IT support, we like the Fortinet FortiGate-60, which receives our Editor's Choice award. This device has an impressive array of features and an intuitive user interface. We're confident that even with minimal IT support, an organization can install and configure the device to provide the protection it needs.

For larger businesses with solid IT staffing, the ServGate EdgeForce has an impressive feature set and solid high-end throughput numbers, earning it second place.

To compare cost, we asked vendors for a quote for a 50-license shop. Of course, pricing will vary for larger or smaller installations.

Right out of the box, the Fortinet FortiGate-60 won us over. It weighs only a few pounds and is not meant for rackmounting, yet it comes with two WAN, one DMZ and four switched LAN interfaces, all 10/100 Mbps. The second WAN interface can be configured for automatic failover.

The FortiGate-60's user interface is intuitive. A setup wizard helped us configure basic features easily. Unlike SonicWall's device, the FortiGate-60 doesn't have a wizard for firewall rules and policies. Even so, the device was a breeze to set up, thanks to easy-to-follow menus. Every time we finished configuring an option, we were certain the function would run as expected. We didn't get this warm feeling from the Astaro or Symantec devices. Although the FortiGate-60 doesn't offer debugging tools from the Web interface, they're available from the CLI (command-line interface).

The FortiGate-60's antivirus implementation is solid; in fact, it's the only device to support antivirus on all the protocols on our wish list. Antivirus configuration was simple. The device let us customize the notification sent after a virus is detected, though it supports only "reject/notify sender" or "reject/notify receiver" when a virus is found. Quarantining viruses is available on Fortinet's higher-end models, starting with the FortiGate-200.Fortinet's antivirus implementation scans for only 3,100 viruses, compared with 60,000 to 80,000 viruses by the other devices. The large disparity has to do with Fortinet focusing on in-the-wild viruses. These are viruses that have been most recently seen in real networks, and don't include viruses for older technology like Windows 3.1. That issue is beyond the scope of this review. However, the Fortinet device did detect all 80 real-world viruses we sent through, and caught MyDoom within 24 hours of its release.

For content filtering, Fortinet uses a subscription service provided by Cerberian. Configuration was simple--we simply logged on to the Cerberian service via the Web, selected the categories of interest and updated a few fields. We were a little surprised that content filtering for FTP isn't supported; neither is content filtering for SMTP. Fortinet says SMTP support will be available in the next release.

Of the devices tested, only Fortinet's and Symantec's provided IDS functionality. Fortinet's doesn't match Symantec's anomaly detection, but the FortiGate-60 did scan for more than 1,400 IDS signatures and, more important for a small business, it actively prevents 34 DoS and intrusion attacks.

For a device aimed at small business, the FortiGate-60 has enough muscle to get the job done. It didn't have the highest throughput numbers, but easily handled our 20-Mbps FTP/HTTP test with up to 12 Mbps of VPN traffic. At around $2,400 for our 50-user scenario, including one year of antivirus and content-filtering subscriptions, the FortiGate-60 is a strong choice for your small-business security needs.

FortiGate-60, $995. Antivirus subscription: $249; content-filtering subscription (50 users): $1,100. Fortinet, (866) 868-3678, (408) 235-7700. www.fortinet.comOf the three Linux-based devices, the ServGate EdgeForce rose to the top. The EdgeForce is a 1U rackmountable box with three interfaces for WAN, DMZ and LAN traffic. Unlike the Fortinet device, it doesn't have an additional WAN port for automatic failover.ServGate doesn't provide wizards for basic configuration, and its interface is neither as slick nor as intuitive as those of the Fortinet and SonicWall devices. Still, the device is fairly easy to configure, and no Linux knowledge is required.

The ServGate antivirus implementation scans for more than 80,000 viruses. When it finds one, it can drop the e-mail, warn the receiver or quarantine the virus. The device has more than 4 GB of space for virus quarantining. It scans for viruses on SMTP, POP3 and FTP traffic, but can't scan HTTP traffic. Spam filtering is available through an add-on--McAfee's SpamAssassin, which we didn't test.

The EdgeForce performed well in our content-filtering tests. We liked that content filtering is supported on FTP as well as HTTP. However, ServGate requires that you set up a Windows server to run the WebSense Enterprise Content Filter package. Configuring this external box wasn't difficult, but it was an extra step. Once this external server was in place, configuring content filtering on the device was straightforward.

The EdgeForce turned in strong performance numbers, outgunning the FortiGate-60 on the high-end tests and easily handling 20 Mbps of HTTP and FTP traffic while processing 20 Mbps of VPN traffic. If you need the extra processing power, for around $4,300 (plus the cost of the Windows server for content filtering) you get a solid security device.

ServGate EdgeForce Integrated Security Platform, device (including EdgeForce Professional module): $1,395. Antivirus subscription: $1,295; content-filtering subscription (50 users): $1,570. ServGate Technologies, (408) 635-8400. www.servgate.com

The Astaro software came installed on a Toshiba server, but it's available as standalone software.We found Astaro's configuration less than intuitive compared with our top two finishers. The device doesn't provide any wizards. But we were able to navigate our way around the software in order to set up the network interfaces and configure the firewall. It helped that we were already familiar with Linux.

The SG30 came with two separate interfaces. We used one for WAN access. The other, which was really six switched ports, handled our DMZ and LAN traffic.

Advanced features were even harder to set up. Antivirus and content-filtering configuration in particular were difficult, requiring some work to understand the options. Rather than having checkboxes called "antivirus" or "content filtering," for example, the software requires that you turn on and configure an HTTP proxy to bring up content filtering and an SMTP relay for antivirus. In addition, we couldn't easily determine if we had configured a service correctly. Once we thought we had configured antivirus correctly and later discovered our mistake only when the device didn't block any viruses.

When configured correctly, the device performed well in our antivirus tests, letting us reject/notify sender, reject/notify receiver, and drop and quarantine e-mail when a virus is detected. Unfortunately, the Astaro antivirus implementation doesn't scan HTTP or FTP traffic.

The Astaro software was the most expensive we tested, thanks in part to some high-end hardware. But unique in this review, Astaro offers a software-only implementation. For around $1,000 plus the cost of the antivirus and content-filtering subscriptions, you can install the software on your own server. Astaro's recommended minimum specifications: a Pentium II 450-MHz with an 8-GB hard drive and 128 MB of RAM.Astaro Security Linux on a Toshiba Magnia SG30, device as tested: $3,795 (software alone: $995). Antivirus subscription (50 users): $695; content- filtering subscription (50 users): $950. Astaro Corp., (781) 272-8787. www.astaro.comSonicWall's excellent user interface was the class of this review. Basic configuration and firewall rules and policies are set up with user-friendly wizards. And though no wizards are provided for configuring content filtering, a wizard is there to help you set up the VPN. We liked that the initial login page provides status information, including warnings and network interface data. Further, SonicWall's logging functionality is the best among these devices--logging is easy to turn on, read and understand.

We did have some problems with the TZ 170 during our tests, however. First, the device had a bug in the firmware implementation of its VPN functionality. SonicWall investigated this and modified its firmware; this fix will ship with future releases. We also had problems with logging when using content filtering: The device's logging feature had trouble keeping up when we tried to access a number of blocked URLs in quick succession.

One disadvantage that hurt SonicWall in this review is that it doesn't support on-device antivirus scanning. Its solution was to have the TZ 170 verify that our client workstations had antivirus software running with up-to-date signature files. If the signature file needs updating, the SonicWall transparently forces the client-side antivirus software to download the new signature file. Although we liked this push feature and believe all client workstations should have antivirus software installed, a multitiered solution that protects both the edge and the workstation is a stronger approach.

SonicWall TZ 170, device: $995. Antivirus subscription (50 nodes): $1,625; content-filtering subscription: $345. SonicWall, (800) 557-6642, (408) 745-9600. www.sonicwall.comIn the area of intrusion detection and prevention, Symantec's device was the strongest, yet also the most complex. In addition to providing intrusion detection for more than 700 attacks, the 5400 performs full application inspection and protocol anomaly detection to look for unwanted traffic actively. This IDS functionality is based on Symantec's ManHunt engine, which identifies new and unknown attacks by analyzing network flows using a technique known as protocol-anomaly detection. Attacks like MyDoom and Code Red are prevented by default.

That said, the Symantec device has a number of problems. Its Java-based user interface is slow and frustrating to use. Its user interface is the least intuitive and usually required us to visit multiple menu options (some many levels deep) to turn on advanced features. Besides requiring additional patience and time, this type of interface may lead to misconfigured features and make you think you're protected when you're not.We also had mixed luck with the 5400 during testing. In the midst of two of our load tests, the device locked up and had to be restarted. Symantec upgraded our software, and the problem seemed to be resolved. However, the 5400 also had the worst score on our content-filtering test. On the other hand, the device performed well on our antivirus test, and its VPN performance was impressive--it handed in the highest VPN throughput numbers.

Symantec wouldn't break down its pricing by device and antivirus and content filtering.

Symantec Gateway Security 5420, device and antivirus and content-filtering subscriptions (50 nodes): $5,278. Symantec Corp., (800) 441-7234, (408) 253-9600. www.symantec.com

HUGH SMITH is an assistant professor in the computer science department at California Polytechnic State University in San Luis Obispo and a member of the Cal Poly Network Performance Research Lab. He was assisted by the NetPRL testing team: Scott Thomas, Brett Tsudama, Mike Watts, Jesse Englert, Mark Porterfield, Matt Mahony and Naris Vipatapat. NetPRL info is available at www.csc.calpoly.edu/~netprl/. Write to them at [email protected].

Post a comment or question on this story.

To see what's available for small and midsize businesses and branch offices that take security seriously despite limited IT and budget resources, we gathered five multipurpose security devices at our partner labs at California Polytechnic State University in San Luis Obispo. We required at least four of these functions: stateful firewalling, on-device antivirus scanning, intrusion detection/prevention, virtual private networking and content filtering.Astaro, Fortinet, ServGate, SonicWall and Symantec sent devices for testing. We divided our testing into stages to put all the functionality through the wringer; for example, for content filtering, we generated a list of 80 porn and gambling URLs to test the vendors' subscription blacklists. For antivirus testing, we e-mailed a stream of 80 viruses through each device and counted the number blocked. We then sent a very large compressed file through each device to verify it wouldn't lock up the antivirus functionality.

At the end of the day, we were pleased with all the devices. However, we considered an intuitive interface a key selling point, and that (with its full feature set) earned Fortinet FortiGate-60 our Editor's Choice award. For a larger organization, take a look at the ServGate EdgeForce, which, though a bit more expensive, had strong throughput numbers and an impressive array of features.Our test plan had three phases.

First, the devices were put on the edge of the Cal Poly NetPRL network. There, each was configured to port-forward SMTP, HTTP, FTP and SSH traffic to servers on the DMZ while allowing traffic from the LAN to the WAN. In addition, we turned on logging and investigated each device's advanced features implementation, including content filtering, antivirus and intrusion detection.

Then we moved the devices into the lab for some performance tests; we monitored throughput and latency and the impact of advanced features on these measurements. In addition, we were "fortunate" to have the MyDoom virus arrive during our tests--24 hours after its release, we ran it through each device to see which vendors had updated their virus definitions.

All four vendors with on-device antivirus caught everything we threw at them, including MyDoom, and handled the large compressed file without difficulty. From a testing standpoint, these four products performed well.

To determine if the devices could carry a substantial load of HTTP, FTP and SMTP traffic, we used an Ixia 1600T traffic generator running IxWeb for HTTP and FTP traffic, and WebLoad for SMTP traffic. We simulated a system with 50 active users--30 HTTP and 20 FTP. Our traffic load was set at 20 Mbps.

In addition, we sent 75 KB of e-mail messages (SMTP) through the devices every two to three seconds. The appliances were configured with firewall rules needed to support the network.

To monitor latency, we used a 4-Kbps HTTP request and response within a four- to five-second period. As a reference point, we measured the latency of this connection without any other traffic on the device. For all devices, this reference latency was less than 3 ms. By watching variations in this connection, we could monitor the effects of various loads on the devices. In the table (at left), a "Pass" means the device was able to handle the offered traffic load with an increase in latency of less than 2 ms on the reference get.

For VPN testing, we provide two results. First, we added 2 Mbps of incoming VPN traffic to our 20 Mbps of HTTP and FTP traffic. We generated this traffic using Ixia's IxVPN and IxChariot software. Then we ran a zero-loss throughput test for VPN traffic only. In this test, we sent short bursts of traffic to the devices and measured their ability to encrypt the data. This zero-loss throughput number gives us an idea of the high-end VPN capabilities of the devices.

Finally, to test each device's content-filtering capabilities, we performed a Google search on the keywords "porn" and "gambling." We then took random URLs from each search and confirmed they truly belonged in the category. After configuring the devices' blacklists to block these URLs, we tried to access the 60 porn and 20 gambling sites, going through the device from the LAN to the WAN. The table below shows the number of URLs that were blocked (denied) out of the 80 sent (higher numbers are better).These all-in-one security devices, while helpful, are not a panacea. Even with a correctly configured edge device performing firewall, antivirus, content-filtering and IDS duties, you need to keep active on your internal security. This includes: