Performance Anxiety

Whenever a new technology hits the streets, the question of whether it can keep up with network traffic quickly follows and with good reason. If a product becomes a network bottle neck or fails to process everything it should be processing, the product has failed. That fear of failure, or perception of being slow, often drives vendors to make optimistic performance claims about their products and drives reviewers like myself put vendors products to the test. This doesn't happen in a vacuum either, IT buyers, whether they are customers or readers want to know whether a product will keep up with traffic. Beware of performance claims about NAC products regardless of the source. Unlike network device testing, there is a whole lot more going on in NAC products that will affect performance and capacity.

In our next round of testing NAC products, I am purposely staying away from performance testing because there just isn't a reliable way to do it meaningfully. Sure, I could toss packets at the NAC devices all day, but meaningful traffic like you would see in a real network is some else entirely.

Designing a performance test, whether you are a vendor or a reviewer, is fraught with decisions and compromises. First and foremost is the realization that there are N+1 test cases in the world that may be interesting to test but there is also a finite amount of time and resources available. Naturally, not every thing gets tested. So a scenario has to be developed that accomplishes a limited set of goals. Sure, the test is contrived to a degree, like all simulated tests are, but a well balanced test tells you something about product performance.

When building a network test, the type of traffic that the device under test (DUT) processes has to be evaluated and created. For a layer 2 switch, it's often sufficient to properly formatted Ethernet frames and the payload doesn't really matter. As the DUT moves up the OSI layer through networking (IP), Transport (TCP/UDP) and beyond, the difficulty in created real traffic rises dramatically. Tools that were designed to test switching and routing will fail to create the protocol stacks to test layer 4 devices like a stateful packet filter firewall. Likewise, the tools that can test a layer 4 firewall often fail at testing an application firewall???a firewall that process, for example, HTTP transactions. Today, it's not hard to generate a large number of TCP sessions and run traffic over them. It is still hard to generate a large number of HTTP users interacting with a web application.

Work-arounds that probably don'tWhen we start talking about NAC, the performance issues are more difficult to test and maybe even more difficult to describe. Of course, the performance issues will vary depending on how the product sits in the network. Not only do the NAC products have to process packets, depending on how they work, they also have to process users and events. A large user population is hard to simulate realistically.I could simply multi-home a host many times using virtual interfaces and assigning each interface an IP address (getting traffic in and out of each virtual interface is a simple but tedious routing problem). That nets lots of IP's and lots of MAC addresses, but it also means lots of identical hosts which may not be useful in a computer or user based test. For example, I can multi-home a Windows computer 20 times, but I would still have one user logged in seemingly 20 times. If a NAC product uses userID or groups to assign policy, or uses persistent or dissolvable agents, well, that won't work well.

I have thought about using virtualization to increase the number of unique computers at my disposal, but even that has limits. Even if I could run say 16 Windows XP virtual computers, giving each a gig of RAM, on a single ESX server, I am not sure they would run particularly well and I know machine performance would nose dive as soon as they started doing disk I/O.

One suggestion that I have heard is to simulate hosts as "guest" computers to get a large population, and then use what ever resources you have to create "real" hosts. At first glance, that sounds like a good idea, but it's not. Based on my 13 years experience of testing products, I know that grossly false test populations typically result in grossly false results. Back in the early 90's when firewall testing was all the rage, people tried to use UDP as "background" traffic because creating enough real TCP from a large number if IP address was difficult. Of course, most network devices like IDS's and firewalls did, and still do, process UDP differently, more efficiently than TCP, so performance numbers for bits per second and number of simultaneous connections were far to optimistic.

My guess is that simulating 100's or 1000's of guest computers???computers that won't be processed by a NAC, will fail to stress a NAC appliance because:

So without a reliable way to load up a NAC system, and I am open to suggestions, performance testing in a lab environment using simulated computers is of limited value. The next best thing is to hear anecdotal evidence of companies that have large user populations to see what their experience is.