Testing To Go

• Cost avoidance: Outsourcing means you don't have to buy all the hardware, software and expertise needed to do worthwhile testing.

• On-demand testing: You can use third parties to meet periodic bursts in demand.

• Knowledge base: By using a third-party testing house during product selection, an organization's in-house lab and operations groups can jump-start their learning and pick up best practices.

• Objectivity: Outsourcing testing is sometimes the only way to get an unbiased, customers' eye view of your organization. Outside-the-firewall, testing services such as Gomez and Keynote Systems can monitor and load actual production environments in the same manner as your customers. This insight, which we think is impossible to reproduce in the lab, can put line-of-business and IT managers in your customers' shoes and illustrate overall service-performance levels.

Other perks can go unnoticed. We chatted with a number of third-party testing labs and found that smart enterprise IT managers are using outsourcing services as political heat shields. An unbiased outsourcer can help relieve pressure on in-house staff.In addition, tight development deadlines can bring significant pressure to release or move new technology to production, often at the expense of adequate testing. Outsourced testing labs can apply the brakes where needed because they're motivated to protect their reputations and business models through careful and accurate testing.

Finally, outsourcing costs can force enterprises to organize and prioritize. When effecting change from within is a nonstarter, an outside perspective might be the push that's needed to reorganize procedures and goals.

Know Thyself

It's important to pick a testing lab with expertise in the technical area you're evaluating. So it follows that you need to know as much as possible about what you're testing.But that's not as easy as it sounds--systems and applications have become more complex because of business, operational and network requirements. This complexity narrows the list of testing labs that will be a fit. We've compiled data to aid you in this selection, but you must ask lots of questions specific to your needs. You want testers experienced in tuning and working with your application or device. Ask how many similar systems the lab has worked on in the past year. In the case of nascent technology, you may need to limit your shortlist to those with experience in the last six months.

The more you know about your environment, the better your chances of finding a good fit. For example, if the hardware under test is to be placed at the edge of your WAN but managed centrally, getting a sense of how WAN connectivity will be simulated in the lab will help you focus your search. Even if you plan to test management over the WAN after lab testing is complete, take the opportunity to discuss your pilot plans and get feedback on the validity of your approach.

To get a handle on your specific needs, gather and prioritize features and problems. Know what bugs are supposed to be fixed by the new version, software, hardware, technology, design or whatever is being tested, then prioritize features to be analyzed based on the needs of your end users.

If you find yourself shopping the third or fourth testing house, be aware that your definitions will have improved, and it may be worth reconsidering first-choice outsourcers using more focused requirements.

Also be prepared to learn from the testing process. For example, it's possible that your new app is well-written and needs only a tweak or two. It's also possible that massive changes are necessary. The outsourcer's testing engineers may have seen your problems before and can suggest different approaches.

One of the most common offerings we ran across when talking to third-party testing labs was analysis of the speed and feed of a device or piece of software. This is the most straightforward type of testing, in large part thanks to testing hardware vendors that have built some useful script-driven bit-blasting boxes.Everyone--we here at Network Computing, enterprises and third-party testing labs--is doing more application testing. However, moving up the stack to test applications is a complex business--we built an entire lab devoted just to this (see inc.networkcomputing.com).

When interviewing a lab, always ask for references. Previous customers can give you insight into test specifics, project management, reporting, knowledge transfer and general engineering expertise.

This last area should also be explored with the testing vendor. Try to meet with the lab's project managers and testing engineers, who should be able to offer specifics about what works and what doesn't. If you've done your homework, speaking with the engineering team will validate and illuminate the specifics of your test.

How skilled are the people? Certifications, while important, indicate only a minimum skill level. Experience in the pressure-packed world of testing matters more. Ultimately, the best way to grade the skill set of a testing house is to hire it for a short-term, tough-deadline test. Short of that, most of the third-party testing labs we talked to were more than happy to open the lines of communication to their project and technical staff, as well as to their customers.

Next, try to determine the employee turnover rate at the lab, especially since the time it performed tests similar to those you need. Ask previous customers the names of the engineers who worked on their projects, and see if those people are still with the testing house. High turnover means lots of experience walking out the door. On the other hand, rotation of various engineering and project managers among departments indicates procedural organization, knowledge transfer and new points of view.

What commitment of resources can be expected? Will dedicated personnel be on your project full time, part time or only as needed? Will they be available for your next release? Having the same test personnel from one test to the next maintains continuity and context.

Try to get a sense of how big a fish you are in the testing house's pond. Is it likely to drop everything for you? Does it have other customers that are about your size, or does it work with your competitors? (The latter might actually improve their testing for your specific situation.) Or are you a small fish that will have to wait for resources or get the second string? Of course, you can--and should--specify contractual deliverables and deadlines with penalties. But that will be little comfort if your testing, and your penalty, is small potatoes compared to the demands of a larger customer.

Word of warning: Enterprises that start with a fixed budget and then refuse to drop portions that don't matter and focus on just the most important aspects of a test will push the outsourcer to reduce quality. It would be a sign of a good testing outsourcer to push back to the point of refusing the job.

• A good test lab will thoroughly scope your project with the intent of setting expectations and prioritizing needs. It can often be more difficult to determine what should be left out of testing than what should be included. Part of the scope will be driven by the cost. Prioritization of needs will help determine what to test and what to leave out.

• The lab should have tests that are repeatable by anyone. No matter its conclusions, all data resulting from tests should be totally objective. When subjective tests are quantified to improve comparisons, those quantifications should be well-documented to improve their repeatability as well as understandability.

To this end, reports and testing documents must explain every step and reason behind each test and result. It's a good idea to ask for some sample reports when evaluating an outsourcer. No matter who reads a report, the resulting data should be clear, regardless of the testing house's conclusion.• The lab should provide frequent project status reports so you won't be surprised by anything at a late date. Ask that the report format and timing meet your internal publication needs.

• Everything the testing lab does for you is owned by you, the customer. This includes documented scripts that are created to run the test. This way, regardless of what happens to the third-party tester, your organization will be able to ensure continuity.

• The contract should include specifics of what is to be delivered and how it is to be turned over. Success in terms of deliverables and a time line should also be defined contractually.

Full Disclosure

Every test should answer four questions: What was tested? What wasn't tested? What succeeded? What failed? But in the case of outsourcing, there's a fifth vital question: Who else does the testing house work for? Getting a straight answer is important as much for reasons of perception as for possible conflicts of interest.Most labs test for hardware and software vendors, as well as for enterprises. Although this is accepted, it carries a negative perception that a product a testing house has previously tested for a vendor customer will do better in a head-to-head test sponsored by an enterprise. Sure, it's possible a testing house could manipulate results to favor a vendor customer. But more often than not, the third-party testing house is painfully aware of your concerns and will have prepared statements about its above-reproach business practices and testing procedures. If you're still not convinced, follow the money. Find out who paid for any test that appears to favor a particular vendor's product or service. Ask to see the testing documentation. Find out how the first four questions were answered.

This odd-bedfellows situation may even regulate the publication of test results. The nature of the vendor relationship, when comparative testing is involved, requires that test results be confidential. On the surface this seems to favor the vendor, but the testing house is protecting itself as well because results that are partially reported or reported out of context can cloud the outcome. Thus, testing houses typically will have a provision requiring that if vendors publish results, the testing house can follow suit. This way, the lab protects its integrity.

Curbing Costs

There's no getting around it: Setting up a complex test is expensive. None of the third-party testing services we spoke with would give us an hourly or daily rate. That's because pricing for most testing engagements is tailored to each specific situation and is based on time and material or a fixed project price. There aren't menus to choose from--such is the nature of any function that requires a custom fit and finish. But there are ways to improve or manage pricing.

Outsourcers constantly have to manage their resources to staff adequately, but the nature of their business is such that jobs are short-run and capricious. If the economy slides, so does development and technology-refresh projects, cancelling or postponing testing. Therefore, testing houses sometimes don't make their numbers at the end of the month. In those cases, there's an opportunity to negotiate a better deal. Play it smart: If you indicate that you're willing to be flexible in your scheduling and know what competitive pricing is, you'll be able to cut a better deal.Also, if your testing is short-term or can be cut into modular chunks, it will be easier for the lab to schedule you between larger, longer tests. This flexibility is valuable and should earn you a discount.

Try to determine the timetable for the setup of your test. If it's straightforward bit blasting for speed-and-feed performance testing, the setup should be simpler than for application or compliance testing. If you know your test fits well into a lab's environment or expertise, you'll be reducing its overhead by leveraging its efficiencies.

So how much should you pay? When doing a cost-benefit analysis to determine whether to outsource or test in-house, it's easy to quantify the hardware and software that's needed; it's knowledge that's difficult. It may be worth outsourcing a few things just to get a sense of the overall process and experience needed prior to budgeting for an in-house testing facility. In addition, you'll need to include soft dollars in the form of labor hours needed to coordinate between operations and development to accurately profile cost.

Time and money are going to dictate, to a large degree, that some features and functions shouldn't be tested. For example, you can save money when testing software by removing regression testing for drivers that have been out and working without incident. Another cost-control technique is to limit third-party testing to mature products that don't need a lot of regression or beta testing. This reduces the scope of the test and the risk of unanticipated delays and cost overruns.

Each test is going to teach you something, and the best way to retain that knowledge is to establish an in-house team that is responsible for maintaining continuity--even if you're not interested in having an in-house test lab. Such a knowledge base will help you maximize both time and cost efficiencies, whether the testing is in-house, outsourced or a little of both.Bruce Boardman, Executive editor of Network Computing, tests and writes about network management and systems. He has 12 years' experience managing networks and distributed computing for a financial service provider. Write to him at [email protected].

Post a comment or question on this story.