Next-Generation Firewalls Put To The Test

How strong is your next-generation firewall? One cybersecurity testing company purports to have the answer.

NSS Labs this week released results of its annual test of next-generation firewalls, comparing 13 NGFWs on performance, security effectiveness, and total cost of ownership (TCO). NSS tested NGFWs from Barracuda Networks, Check Point Software Technologies, Cisco, Cyberoam, Dell, Forcepoint, Juniper Networks, Hillstone Networks, Huawei Technologies, Palo Alto Networks, and WatchGuard Technologies.

NSS's test reports quickly led to some real-world results. Cyberoam's CR2500iNG-XP v10.6.3 did not perform well when under extended attack during NSS's tests, simultaneously failing to block malicious traffic and let legitimate traffic through. This made the CR2500iNG-XP a security outlier, scoring 58.1% in security effectiveness and 60% in stability and reliability, whereas 11 of the other 12 NGFWs scored above 90% on security effectiveness and all 12 scored 100% on security and reliability.

Accordingly, Cyberoam cleaned up its act. The day after releasing its first report, NSS blogged that Cyberoam had submitted a software update resolving the CR2500iNG-XP's stability and reliability issues. The bump in the product’s stability-and-reliability score combined with retesting of its performance and exploit-block rate boosted its security-effectiveness rating to 96%, putting it at tenth place -- just above the 95.9% Palo Alto's PA-7050 earned.

There is more to NGFW shopping, however, than selecting on security effectiveness alone, NSS cautions.

"There is frequently a trade-off between security effectiveness and performance," NSS wrote in its testing report. "Because of this trade-off, it is important to judge a product’s security effectiveness within the context of its performance and vice versa."

firewall-29940_640 (1).png

For its part, Palo Alto's PA-7050 took top honors on performance tests, performing more than 75% better than the average connectivity rates of its 12 competitors. Cisco's FirePOWER Appliance 8350 -- the PA-7050's closest connectivity competition -- performed only about two thirds as well as the PA-7050 on connectivity. The PA-7050 also demonstrated extremely high throughput rates during testing, with only modest latency.

However, Fortinet's FortiGate 3200D -- which tied with Check Point's 13800 NGFW Appliance by scoring 99.6% for first place in NSS's overall security-effectiveness assessment -- was the top performer in a special "real-world protocol mix" test designed to mimic the traffic of a real-life data center. Palo Alto's PA-7050 performed only moderately well here.

NSS reports that WatchGuard's XTM 1525 had by far the worst connection rates while demonstrating considerably low throughput rates and, at times, high latency. Moreover, NSS scored the XTM 1525 at 87.7% in its security-effectiveness tests, dropping it to last place since Cyberoam's software update.  On the other hand, WatchGuard’s XTM 1525 reportedly has a much lower TCO than Palo Alto's PA-7050. NSS estimated the PA-7050's TCO to be $31 per protected Mbps, while calculating the XTM 1525's TCO to be $18 per protected Mbps.

Indeed, the PA-7050 has by far the highest purchase price (a whopping $10,263,800) of the tested NGFWs, but it is apparently not the most expensive in the long term. That dubious honor belongs to Juniper's SRX5400E -- with a calculated TCO per protected Mbps of $97. The lowest calculated TCOs came from the two Chinese-manufactured NGFWs: Hillstone's SG-6000-E5960 and Huawei's USG6650 came in at $6 and $7 per protected Mbps respectively.

NSS put its "Recommended" stamp on seven of the 13 NGFWs it tested, indicating that they tested better than average on both security effectiveness and TCO:

What are your thoughts? Do you use any of these NGFWs or another that NSS didn't test? Tell us in the message board below.