AntiSpam Techniques

Which brings us to the obvious question: If everyone hates spam so much, why is it one of the largest growth industries in the world? Answer: Because people do make money by inundating us with advertisements for junk. As long as one sucker per 100,000 recipients responds to the "click here" or "call this number" portion of the spammer's message, there is sufficient incentive for sending out an additional 5 million messages.

It continues to amaze us that anyone could be dumb enough to respond to this stuff. Because antispam vendors have become adept at blocking simple spam, spammers have adopted tactics so bizarre that getting even one response in 1 million seems unlikely. E-mail message subjects regularly contain words that aren't words, gross misspellings, symbols that you'd normally find only in math equations, poor grammar and a host of other miscommunications that would typically render any message that followed completely suspect. Recent examples from our inbox include such memorable subject lines as "Re: legate enol," "sku1per via1hgra" and our personal favorite, "Give me some money, please." But still, numbskulls click and call and encourage and keep the spam industry alive.

The fight against spam is being waged on two fronts, legal and technological. We hear from time to time about small claims and spectacular victories in the courtroom, but we believe--as do a majority of our antispam poll respondents--that legislative efforts alone will not eliminate spam. Only 11 percent of our 455 qualified respondents think legislative efforts are even somewhat effective deterrents to spam, and fewer than one in four holds out hope for more effective legislation in the future.

Legal challenges against spammers are complicated by three obstacles: tracking down the source of spam, identifying who the spammers really are, and dealing with international boundaries when attempting to prosecute identified spammers. Many IT people mistakenly think that most spam originates overseas and that U.S. legislative efforts would be effective against only a small portion of spam. But in February 2004, Sophos, an antivirus software provider, traced the origin of all spam received by its research center over a two-day period and found that nearly 60 percent was sent from within the United States.So the CAN-SPAM (Controlling the Assault of Non-Solicited Pornography and Marketing) Act of 2003 (see ID# 1501buzz2) should be effective, right? Nope. Of the spam that's sent from within the States, between 30 percent (Sophos estimate) and 70 percent (according to MessageLabs, a British antispam service provider) is sent using computers that are infected with spam-relay Trojans and worms. These programs allow spammers from anywhere in the world to relay their messages through thousands of infected systems without the owners' knowledge.

Still, the Federal Trade Commission filed criminal and civil charges against four named defendants on April 29 for violating provisions in the CAN-SPAM Act. This marks the first government case against spammers based on the new law. If the government's case is successful, we're likely to see a number of additional government cases filed in the months to come, and that's good news.And in March, four U.S. firms--AOL, EarthLink, Microsoft and Yahoo--filed six lawsuits in four federal courts against hundreds of spammers using provisions in the CAN-SPAM Act. Emphasizing the difficulties inherent in identifying spammers, only three defendants were identified by name in the lawsuits, while more than 200 were tagged as John Doe.

U.S. courts are the main venue for the big e-mail service providers' battles against spam. But legal proceedings are expensive; the process to identify John Doe defendants is a massive undertaking. And, like all things legal, these cases will take time, maybe years, to wend their way through the courts. If they're successful, we'll all benefit from these providers' efforts to shut down the big spam houses. But the vast majority of companies don't have the resources to wage this kind of war, and we don't need results at some unknown point in the future--we need relief today. So, for the rest of us, technology holds the key.

Technology hasn't conquered spam yet, but 70 percent of those responding to our poll said that antispam technology is at least somewhat effective, and nearly a quarter rated it as very effective. And almost 83 percent think the technology will get even better.

There's good reason for this optimism. Based on the technical answers available and the number of vendors developing new methods and combinations, the most difficult part of implementing an antispam solution may be parsing the choices. Our advice: Concentrate on the key metrics of accuracy and cost to narrow the field, then pilot-test each system to determine which is best for you. To help you make your choice, we invited 35 (yes, 35) vendors to participate in an accuracy shoot-out in our Syracuse University Real-World Labs®. The results of that testing and our analysis of the Top 10 vendors' features starts here.

From a technical perspective, accuracy is fundamental. Products that block the most spam without misidentifying legitimate mail (aka ham) as spam should move to the top of your short list of vendors. All the vendors publish their own accuracy ratings, but you'll need to make sure their definition is the same as yours. A simple formula takes the total number of mistakes a product makes when classifying mail--both spam classified as ham (false negative) or ham classified as spam (false positive)--and divides by the total number of messages.

But this formula doesn't take into account the fact that false positives cause more trouble than false negatives--a false positive usually means a person doesn't know he or she has received a legitimate e-mail, because it's stuffed in a spam folder somewhere or, worse, deleted sight unseen. So we developed for our testing a weighted accuracy rating that treats false positives as five times more expensive than false negatives. Plug in any number you like or treat false positives and false negatives the same, but just make sure your definition is rigorously applied so you're comparing apples to apples.The price you must pay to rid yourself of the spam menace is the other main consideration. If you're like us, you hate to spend even an iota of time or money on a technology problem that isn't of your own making and that won't add a single hard dollar to your bottom line. But there's no way around it, so consider implementing an antispam solution as cost avoidance. Your goal should be to buy the best product for the job for the least money possible, and limit the resources you devote to implementation and upkeep. Prices for the products we tested ranged from $0.27 to $34.13 per user per year, and we found that more money doesn't necessarily buy greater accuracy.

If you have the time and technical chops to set up a system and run it without formal support, open-source antispam offerings are available. We recommend SpamAssassin (au.spam assassin.org) and SpamBayes (spam bayes.sourceforge.net).

Time spent researching and testing before implementation is time well spent--if you're focused on what's important and have a good understanding of your needs. Here are some questions to consider:» Is e-mail so critical to your business that any mail lost because of an incorrect spam classification would create a large financial risk? If so, make sure that the product you select treats every e-mail as if it were gold and that your users will always be able to easily search a mail quarantine area or spam folder.

» Do you want to set up aggressive antispam thresholds for some users and more lenient thresholds for others? Then the product you select must support multiple profiles and make it easy to manage which user belongs to which group.

» Do you want users to set their own thresholds for spam, build whitelists and blacklists for senders, define their own custom rule sets and determine how the product handles mail identified as spam? Or do you want the product to be centrally administered in a one-size-fits-all manner?

» Are you in a business, like pharmaceuticals or banking, where discussions about cheap medications or low mortgage rates are an important part of your day-to-day discourse? Choose a vendor that handles these topics with intelligence and a product you can customize.

» Do you want a product that also scans incoming mail for viruses?Note that antispam offerings are currently available in three flavors: software you install on your own hardware, appliances that ship as turnkey systems, and hosted solutions that don't require any investment in hardware or software and provide a fixed per-user cost. According to our reader poll, 68 percent prefer an in-house setup, while just 7 percent would consider a hosted service. We suspect that security concerns surrounding corporate e-mail being processed by a third party has something to do with this low figure. However, our experience with the outsourced vendors in our review was positive, and we wouldn't hesitate to recommend them as partners.

On a side note, 18 percent of those surveyed said they would consider a desktop antispam strategy--one that's integrated with Outlook or Notes, for example. There are a couple of reasons we don't view desktop solutions as viable for corporate companies. When antispam filters are implemented at endpoints rather than the Internet perimeter, your mail server has to process and store each piece of spam as if it were legitimate mail, which dramatically increases the load on the server. Even more important, support and management for desktop antispam implementations is much less favorable than for a centralized system. One exception: If you're a small company that outsources your e-mail services, then desktop offerings are worth investigating.

We sometimes devote a lot of space in these pages to complex ROI (return on investment) formulas or TCO (total cost of ownership) figures to help you justify spending money on a particular technology to solve a business problem. In this case, we feel confident that you're already convinced, but take a good look at our "Spam By the Numbers" chart. Here are some additional numbers to throw around, just for fun.

In 2003, a typical 10,000-user organization needed 21 Exchange servers to handle all of its e-mail, both ham and spam, according to the Radicati Group. Because 24 percent of the mail was junk, five of these 21 servers were needed just to handle spam, and that number isn't going down. Additional servers mean additional costs for hardware, operating systems, applications, maintenance, administration, upgrades, machine-room space, utilities, operations staff, network infrastructure and archiving, plus an increased likelihood of unscheduled downtime caused by increased complexity.

This is not a pretty picture, but it's an unavoidable one if spam goes unchecked in your organization. If you don't do something, spam will overwhelm your infrastructure and your users. An antispam strategy is a lot like security: You really can't do without it.

RON ANDERSON is NETWORK COMPUTING's lab director. Before joining the staff, he managed IT in various capacities at Syracuse University and the Veteran's Administration. Write to him at [email protected].

You don't need us to tell you that spam volume is reaching critical mass. The deluge is growing at a rate that's dwarfing even the most dire predictions. And there are security implications as well, from zombies to spam-borne nasties like the recent Osama Trojan. So the question is: How do we crack down on junk e-mail?

In "Sick of Spam?" we outline two approaches, legal and technological. Call us cynics, but we don't have great hopes for the CAN-SPAM (Controlling the Assault of Non-Solicited Pornography and Marketing) Act of 2003. Although we're rooting hard for the large ISPs that are hauling hundreds of junk e-mailers into court, spammers are like roaches: Crushing a couple is satisfying, but there are 100 more where those came from. We need relief now.

That's where technology comes in. Antispam vendors feel our pain, and they're working hard on new products and services. In fact, when we announced that we were planning to test spam filters, we were overwhelmed by the response. Thus was born the mother of all antispam product reviews.The methodology we used for these tests is a departure for us because we used the results of our accuracy shootout to select the vendors that would participate in the full review. And now, 35 invitations, 27 entries, 23 usable results and 10 finalists later, we present our assessment of the top offerings in this market based on accuracy tests, price and manageability. The winner: Barracuda Networks' Spam Firewall. Its combination of features, performance and price earned it our Editor's Choice award.