Ultimate anti-spam guide: 11 products tested

ContentsBitDefenderClearswiftCA eTrustGFIIronPortMailGuardMcAfeeMessageLabsNetIQNetwork BoxSymantec BrightmailEditor's ChoiceAbout RMITHow we tested special report From server-level software, to appliances, to managed services, we find what solutions are available to help enterprises manage the onslaught of unsightly spam.It has been over a year since we reviewed anti-spam offerings.


Contents
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

Technology & Business magazine


special report From server-level software, to appliances, to managed services, we find what solutions are available to help enterprises manage the onslaught of unsightly spam.

It has been over a year since we reviewed anti-spam offerings.

Back in those days there were few enterprise-level solutions available to deal with this issue. Since then the market has literally exploded -- from four or five popular applications on the market last year to a submission of no less than 11 for this review.

And even still there were some notable names missing like Sophos, Surfcontrol, and Trend Micro. Both Surfcontrol and Trend Micro are in the midst of changing their older applications over for newer/updated versions and were not currently in the position to submit. Sophos on the other hand, while we would have loved to have squeezed them in somewhere with the other 11 products, unfortunately responded too late to get included in the review. When evaluating anti-spam products don't forget to consider these three vendors also.

This review is more of a guide to the current state of play in the world of annoying spam e-mail. For this roundup, we looked at each vendor's product based on common criteria such as installation, configuration, and administration. We did not perform any "official" accuracy and performance testing on the products. We set the programs up in modes to test both controlled and live messages, however the results of these brief tests would just add more confusion to the mix than anything and certainly didn't show any unexpected results.

Editor's note: This report was first published in Technology and Business magazine. Due to space constraints, the section explaining how the products were tested was omitted. However, a full explanation on testing procedures has been posted online and can be found here.

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

BitDefender v1.9 for MS Exchange 2003

BitDefender is a software application, running on Microsoft Exchange 2003. The installation was the easiest of all the products reviewed, taking less than a couple of minutes to run up with very little administrator interaction. In fact, even the initial configuration is mostly done for the operator also. Further configuration can be handled through the intuitive user interface.

Administration and reporting was a little on the lighter side when compared with some of the other applications reviewed here, however if simplicity is the requirement and the environment is Microsoft Exchange 2003 then BitDefender is worth a look.

The only application in this review specifically for Exhange 2003. Most of the other applications can be setup to act as gateways for almost every flavour of mail server available.

Click to enlarge
Product BitDefender v1.9 for MS Exchange 2003
Price AU$315 incl GST for 10 users
Vendor NetFreighters
Phone 1300 304 007
Web www.netfreighters.com.au
www.bitdefender.com
 
Interoperability
½
Only works with MS Exchange 2003.
Futureproofing
A good range of features and adequate reporting is provided.
ROI
½
Excellent pricing.
Service
½
Excellent support options.
Rating
½

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

Clearswift MIMEsweeper for SMTP 5.0

MimeSweeper is another of the software applications that were submitted for this review.

The installation was potentially the most difficult of all the products reviewed here, possibly due to the level of complexity, features, and scalability built into the MimeSweeper product.

We say potentially the most difficult because the development team at Clearswift have certainly done their homework on this one and the installation routine virtually takes the installer step-by-step through the whole setup phase with a minimum of fuss.

And when you see how many third-party applications, plugins, and basic requirements this application requires to run you will be amazed.

Firstly the .NET framework from Microsoft needs to be installed, this is bundled with the MimeSweeper distribution, then the installer scans the system for missing components, on our machine it found that we had not installed MSMQ (Microsoft Mailer Queue) and that there was no ASPNET user in our Active Directory.

The first one was a no-brainer -- just install MSMQ from the 2000 Server disc, but the ASPNET user nearly stumped us as technically, it is supposed to be installed when ASP is installed with IIS, however Clearswift knew there would be people like us trying to bend the rules and install our Server O/S, Active Directory, Exchange, and anti-spam apps all on the same machine despite the security risks associated with having AD on a relatively publicly accessible machines -- all good security administrators and the fellows at Clearswift will tell you this is definitely a no-no.

However, seeing this machine would be up for only a matter of hours in a test environment we wanted to plough ahead, and when we delved into the readily available linked help pages, the application pointed us in exactly the right direction (to a Microsoft Knowledge Base document detailing this exact issue and pointing out in no uncertain terms the potential risks and hazards associated with putting all our precious eggs in one basket).

Once we followed the Microsoft document through the steps to create a secure-as-possible ASPNET user on the machine and clicked refresh on the "required components" section of the MimeSweeper installer, the system was rechecked and we were able to continue with the install. No less than four more add-ins were required after this, thankfully all included with the distribution. These were MDAC v2.8, MSDE 2000 SP3, .NET v1.1 SP1, and Microsoft Visual J .NET 1.1.

Again the installation manager for each of these was nothing short of brilliant and after a required reboot the rest of the actual MimeSweeper installation could be completed, included in this was a wizard which set up most of the initial configuration of the system for us. Finally a last reboot and we were good to go.

Administration is via four centres, these are centrally managed via one main interface. Overall the administration is very well laid out, easy to use, and extremely well documented.

At the end of the day, one should only have to install an application of this caliber once or twice in the enterprise to cover most SMTP mail servers, therefore while we harped on a bit about the installation routine involved it was mostly an exercise in pointing out how if the installation routines and installer applications were not as well designed as they were it really could be a complete nightmare. It goes to prove that the engineers at Clearswift have really got a grasp on their art. Turning something that could be so potentially horrendous into a usable form really deserves some recognition.

Click to enlarge
Product MIMEsweeper for SMTP 5.0
Price AU$3000 for 50 users
Vendor Clearswift
Phone 02 9424 1200
Web www.clearswift.com.au
 
Interoperability
Works with any mail server.
Futureproofing
Very good range of features provided and good reporting.
ROI
Very well priced.
Service
A little light on warranty (90 days/30 days) when compared with other anti-spam vendor warranties.
Rating
½

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

CA eTrust Secure Content Manager v1.0

Once the installation routine is commenced the operator is given the choice of setting the station type, either Master or Slave, and then selecting the type of scanning required by the application, either HTTP/FTP, or SMTP, or both.

Naturally, while this application has the added feature of being able to act as a content filter as well as a SMTP filter we chose SMTP as this is a anti-spam review not a content filter review.

Once a few more quick wizards are gone through the application is installed and ready to go. Overall the installation and base configuration is very straightforward.

Interestingly, this application is designed to run in a distributed environment and can report back to a central machine which would gather information from a number of installed nodes on the network, making it a boon for those with plenty of remote offices.

Likewise the quarantine manager can be centralised on a separate system making the mail administrators role easier too.

Overall a neat application definitely suited to remote/distributed environments. Again CA have used a familiar interface and adopted a very straight forward approach.

Click to enlarge
Product eTrust Secure Content Manager 1.1
Price US$55 per node (for full suite)
Vendor Computer Associates
Phone 1800 999 985
Web www.ca.com/etrust
 
Interoperability
Works with any mail server, designed to work easily over several mail servers.
Futureproofing
½
A good range of features and good reporting is provided.
ROI
½
Average.
Service
Very goodââ,¬"12 months warranty included.
Rating

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

GFI Mail Essentials v10.1

One of the six software applications received, version 10.1 of GFI's Mailessentials application was one of the easiest of the "gateway" programs to install and configure.

Asking very minimal input from the operator, the application virtually installed itself including the required MSMQ if it had not already been installed on the server.

There are two main programs associated with Mailessentials -- GFI Mailessentials configuration and GFI Mailessentials monitor.

The administration takes a little while to get used to however once a operator has spent some time with the interface it becomes very easy to drive. The monitor application is very good giving a brief overview of the current state of play in relation to the system.

In conclusion a very straightforward down to earth application with many advanced "behind the scenes" features. This should be on a shortlist for anti-spam evaluations.

Click to enlarge

Product GFI Mail Essentials for Exchange/SMTP v10.1
Price From AU$550 for 25 users
Vendor GFI Software
Phone 1800 22 55 43
Web www.gfi.com
 
Interoperability
Works with any mail server.
Futureproofing
½
A good range of features and good reporting is provided. We would recommend looking at the MailSecurity Bundle option to increase functionality.
ROI
Very well priced without the MailSecurity option, and with MailSecurity the price is average.
Service
No warranty details provided.
Rating
½

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

IronPort C30

Of three appliances submitted in this round-up, Iron Port is certainly the big brother of the systems. Sporting redundant power supplies, hard drives, and dual processor capabilities in the higher-end models, this is a purpose-built messaging gateway, with extra security features that simply would not be possible to integrate into a software application.

Naturally, because of the extra features and the hardware involved, this solution comes at a premium price, however if security is your aim and you would prefer not to have your mail servers in the front line, then our suggestion would be to evaluate one or more of the three appliances here.

Initial installation and configuration takes approximately 15 to 20 minutes for a total newcomer to the equipment, however once experienced this could probably be completed in less than 10 minutes.

For those of you who have had experience with IronPort appliances, the latest versions of the IronPort operating system now come with a graphical user interface (GUI) that steps the operator through each step of the initial configuration. For the die hards out there don't despair you can still access all the features and functionality you need with the command line interface (CLI) -- it hasn't disappeared.

Due to the interface configurations the mail servers can be totally shielded from the rest of the world. There are also many advanced features that operators can select to use which adds up to great security, granularity in configuration, and flexibility.

Overall, while not the cheapest box on the block the IronPort systems are still definitely one to put on the shortlist, particularly for larger organisations or ones requiring a high level of security.

Click to enlarge
Product IronPort C-Series mail gateway appliances models C10, C30, C60
Price AU$18,000-AU$200,000+
Vendor IronPort Systems
Phone 02 9943 1860
Web www.ironport.com.au
 
Interoperability
Works with any mail server.
Futureproofing
½
Very good range of features provided and very good reporting.
ROI
½
If the business has at least 250+ mailboxes to protect then the pricing is attractive.
Service
12 months warranty.
Rating

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

MailGuard

The first of the managed services -- and we must admit we are becoming fans of these services.

Once the pages of the documents have been completed and sent off, one waits until the welcome e-mail pack arrives in the inbox, all that is left to do is follow the instructions to reconfigure the MX records for the domain(s), and tighten up the security and configuration of the mailserver.

Once completed sit back and watch the filtered messages come in a go out. It really is as simple as that.

Administration is handled online via a series of Web pages that are very logical and well layed out.

In conclusion, if you want to take the stress out of managing spam and have someone else do the work then a managed service is the way to go for sure.

Click to enlarge
Product MailGuard
Price AU$199 setup, AU$5.50/seat
Vendor MailGuard
Phone 1300 30 44 30
Web www.mailguard.com.au
 
Interoperability
Works with any mail server.
Futureproofing
Very good range of features provided and good reporting.
ROI
Very well priced.
Service
½
Unlimited warranty is very good.
Rating

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

McAfee SpamKiller & WebShield

McAfee has both bases covered with software and hardware e-mail filtering solutions in their stables. For this review, instead of adding yet another software product we selected one of the McAfee appliances.

For interest's sake we also ran up their software package and were pleasantly surprised to find that both technologies employed virtually the same user interfaces, therefore for large or distributed organisations looking for large and small e-mail servers to be covered McAfee may well have a scalable software or hardware solution to suit with relatively unified interfaces. Certainly something to consider.

Installation of the appliance once connected up and turned on is very simple, taking no more than 15 minutes to configure. Multiple network interfaces provide an added level of security and advanced routing in the configuration allows quite a number of different permutations to be used depending on the environment the equipment is being deployed in.

There is a very good amount of logging and reporting provided as well as other features such as content filtering and antivirus protection.

Overall a very well designed robust appliance which would suit most SMEs looking to remove their mailserver from the frontline.

Click to enlarge
Click to enlarge
Product McAfee SpamKiller for McAfee WebShield 3000 series appliances
Price 3110 AU$1996, 3200 AU$6935, 3300 AU$13,895.
Vendor McAfee
Phone 1800 644 646
Web www.mcafee.com.au
 
Interoperability
Works with any mail server.
Futureproofing
½
Very good range of features provided and good reporting.
ROI
½
Attractive price for 100+ mailboxes.
Service
12 month warranty.
Rating
½

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

MessageLabs AntiSpam Service

The second of the managed services goes like this: we (as per the MailGuard write up) complete the form, send it in, receive the welcome kit, follow the instructions regarding DNS and Mail-Server, see the filtered mail arrive.

All the customisation and administration is handled via a Web interface.

If you are comfortable with routing your MX records to a third party for filtering then I would say give it a trial and see how simple it is for yourself.














Click to enlarge
Product MessageLabs AntiSpam Service
Price Per user, per month -- price not supplied
Vendor MessageLabs
Phone 02 9409 4360
Web www.messagelabs.com.au
 
Interoperability
Works with any mail server.
Futureproofing
Very good range of features provided and good reporting.
ROI
Price not supplied.
Service
½
Warranty and service is included with the managed service subscription.
Rating

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

NetIQ MailMarshal SMTP 6.0.3.8

NetIQ was the belle of the ball last year in the anti-spam reviews that we performed and this version is also no slouch with some impressive improvements.

The installation is launched from within a local Web page which also provides a plethora of other documents and options.

Once the install routine starts, the system installs MSDE 2000 and reboots the machine, once rebooted and installation continues, the user is guided by a wizard which completes the initial configuration.

Not only can MailMarshal be run in conjunction with another e-mail server package such as Exchange it can also operate completely as its own e-mail system.

Further configuration and administration are simple to achieve with a separate MailMarshal configurator application and MailMarshal admin console.

For a SMTP gateway application, indeed as a stand alone mail package for SME's give NetIQ an evaluation.

Click to enlarge
Product NetIQ MailMarshal 6.0.3.8
Price 25 users AU$1155, 50 users AU$1850, 100 users AU$3325, 250 users AU$6925, 500 users AU$12,000
Vendor NetIQ
Phone 02 9959 2313
Web www.netiq.com.au
 
Interoperability
½
Works with any mail server. Even includes an integrated mail server.
Futureproofing
A good range of features, and excellent reporting.
ROI
Excellent pricing, however maintenance is not included.
Service
No warranty details provided.
Rating

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

Network Box Internet Threat Prevention System

The third managed service submitted for this review has a twist, it is an appliance which is delivered, installed and configured by a Network Box engineer.

Therefore the hardware is on your own premises, not that you have to do anything with it particularly, except for look at the neat status LCD on the front of the 2U chassis, it is still remotely managed by Network Box's engineering team.

There are five different options available when it comes to the size and scale of the Network Box lineup. If you are hovering between an in-house managed system and a outsourced managed service, then perhaps Network Box can provide a solution for you.

Product Network Box Internet Threat Prevention System
Price AU$270/month
Vendor Network Box
Phone 03 8841 0000
Web www.network-box.com.au
 
Interoperability
Works with any mail server.
Futureproofing
Very good range of features, and good reporting.
ROI
½
Attractively priced for 50+ mailboxes.
Service
On-site replacement for the duration of the managed service is excellent.
Rating

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

Symantec BrightMail AntiSpam 6.0.1

The last but certainly not least software application in this anti-spam review is Symantec BrightMail 6.0.1.

This software is very modular, the administrator must first install a scanner on the target machine(s), then once licensed and the machine restarted, the Brightmail contol centre can be installed.

Part of the installation loads and configures Tomcat and MySQL. Following the completion of the installation the administrator can launch the browser-based interface for the control centre, add whichever scanners were deployed and fine tune the applications configuration.

Brightmail updates very frequently and its summary page shows reports and graphs for recent activity. Antivirus is also an option.

Possibly one of the easiest to install and get up and running with excellent reporting.

Click to enlarge
Product Symantec Brightmail Anti-Spam v6.01
Price AU$19 per user, and less
Vendor Symantec
Phone 1800 000 423
Web www.symantec.com.au
 
Interoperability
Works with any mail server, designed to work easily over several mail servers.
Futureproofing
½
Very good range of features and excellent reporting.
ROI
Appears quite good.
Service
No warrant details provided.
Rating
½

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

Editor's Choice
Editor's Choice: Symantec Brightmail, Network Box, and Iron Port

We have split the Editor's Choice into the three separate anti-spam technology categories; software, managed service and appliance. Because there are so many e-mail environments out there, most administrators looking to deploy an anti-spam solution should by now have a fair idea which type of technology would suit their requirements.

Software winner: Symantec Brightmail, for ease of installation, configuration and administration as well as an excellent user interface and detailed -live" graphical reporting it would be hard to surpass these features.

Managed Service winner: Network Box, if security is a concern then Network Box has the bases covered, if availability and redundancy are your preferred choice then a trial of either MailGuard or MessageLabs may be on the cards.

Appliance winner: IronPort, strong security, redundancy and recently developed ease of installation with the new GUI make this appliance the choice in this review. For those with a tighter budget then perhaps one of the McAfee WebShield appliances may be considered and are still very worthy contenders.

Please note that these decisions were not based on accuracy testing. Given the Labs' extensive anti-spam testing experience, when being initially trialled/evaluated we would expect most anti-spam applications to run around 65 percent to 70 percent spam catch accuracy with very low to zero false positives in "default" or "out of the box" configurations.

Then, once given the benefit of being "tuned" or "tweaked" and having localised white and black lists applied they should run at about 85 percent to 92 percent. Remember each e-mail server environment and organisation's requirements are different therefore it would not be fair to try and compare the results based on limited tests and scenarios it's far beyond the scope of this review.

Product & version BitDefender v1.9 for MS Exchange 2003 MIMEsweeper for SMTP 5.0 eTrust Secure Content Manager 1.1 GFI MailEssentials for Exchange/SMTP V10.1 NetIQ MailMarshal 6.0.3. 8 Symantec Brightmail anti spam version 6.01
Company Softwin / NetFreighters Clearswift Computer Associates GFI Software NetIQSymantec
Price AU$315 for 10 users 50 users AU$3000 incl. 1 year support US$55 per node (for full suite) 25 users AU$550, 100 users AU$1075, unlimited users AU$2150. 25 users AU$1155, 50 users AU$1850, 100 users AU$3325, 250 users AU$6925, 500 users AU$12,000 AU$19 per user; volume discounts available
Warranty 30 days trial 90 days software CD / 30 days installed software 12 months N/A N/A N/A
Operating System   XP Professional, Win2k server/advanced server or Windows 2003 Windows 2003 Server Standard Edition, Windows XP Pro with SP1, Windows 2000 with SP2, SP3, or SP4 Any Windows 2000/2003/XP operating system. Windows 2000 or Above Microsoft 2000/2003 Server, Solaris 8 or 9, Linux RedHat ES/AS 3.0
E-mail server software Microsoft Exchange 2003 Server Any - Any. Extra functionality in AD environments. Any SMTP Mail Server Exchange / Notes Foldering agent
Network requirements   TCP/IP TCP/IP SMTP Hardware Sizing dependant on Email Volume minimum 100mbit lan
Support for address whitelists (Y/N) Y Y Y Y and Autowhites addresses based on recipient address of outbound email. Y Y
Support for address blacklists (Y/N) Y Y Y Y as well as DNS blacklists. Y Y
Support for keyword blocking (Y/N) Y Y Y Y Y Y
Support for embedded URL blocking (Y/N) Y Y Y Y Y Y
Support for attachment scanning (Y/N) Y Y Y N Y Y
Support for embedded image scanning (Y/N) N Y N Y N Y
Active HTML detection & blocking (Y/N) Y Y Y N Y Y
Integrated e-mail anti-virus (Y/N) Y N Y N Y Y
Type of scanning employed heuristic,w/b lists, charset filter, url filter, keyword filter N/A AV, spam, URL filtering, keyword, profanity, file type, virus hoaxes, activex and Java applets Bayesian, SPF, directory harversting, DNS Blacklist, blacklist, header checking and keyword checking. Lexical Analasys Reputation filters, URL filters, signature filters, heuristic filters, content filters, AV filters
Automated definition updates (Y/N) Y Y Y Y, for the Bayesian engine. Y Y

This article was first published in Technology & Business magazine.
Click here for subscription information.

Continued ...


Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT
How we tested

About RMIT IT Test Labs

RMIT IT Test Labs
RMIT IT Test Labs is an independent testing institution based in Melbourne, Victoria, performing IT product testing for clients such as IBM, Coles-Myer, and a wide variety of government bodies. In the Labs' testing for T&B, they are in direct contact with the clients supplying products and the magazine is responsible for the full cost of the testing. The findings are the Labs' own -- only the specifications of the products to be tested are provided by the magazine. For more information on RMIT, please contact the Lab Manager, Steven Turvey.






Contents
Introduction
BitDefender
Clearswift
CA eTrust
GFI
IronPort
MailGuard
McAfee
MessageLabs
NetIQ
Network Box
Symantec Brightmail
Editor's Choice
About RMIT

How we tested

To do a complete and thorough live accuracy test would take at least two to three months. It would involve the setting up of two concurrent running mail servers and applications per product on test, one set to default vendor baseline static -out of the box" settings and the other as a dynamic -tweakable" system to ensure that benefits were being derived on a day-to-day and week-to-week basis between the static and dynamic machines. So for the 11 products in this review, 22 mail servers would be required.

We would then have to select the combination from our live honeypot domains which would provide the best mix of unique spam messages for the testing.

Honeypots are live mail servers with valid domains and user accounts that we constantly have running, which attract and collect spam. It takes considerable time to build these honeypots up as you can't simply subscribe or add your e-mail address to the spammers database -- then you are inviting or entrapping the spammer and those messages would have to be classed as -grey" not -spam".

We have to ensure that our test e-mail accounts are harvested via normal spammers means, like domain and address harvesting from live Web sites, name database additions etc. Our honeypots have a history of almost three years now.

Once we had our live spam feed we then need to inject a live ham/grey mail feed too. We have modified a centralised mail server to enable us to perform this initial combination then aggregation of the message stream to ensure that each mail server receives exactly the same feed live from the Internet to one address on the mail server as though the message had come straight from the source to the final destination.

This is very difficult to achieve particularly considering that many anti-spam applications rely on the original e-mail header information being intact. Something that mail applications like Novell Groupwise, Microsoft Exchange and Outlook do not do.

So while this is all good and well, we would then have 22 mail servers and anti-spam applications up and running with live feeds of spam, ham and grey mail. We would then set up a machine or groups of machines to POP the e-mail messages from the respective servers using Outlook Express. OE keeps the headers intact for later reference or use via our controlled/static test, yes once the live testing is over we have developed a methodology for mass -re-testing" under a controlled environment.

Once the messages are in OE the hard part starts sorting out the missed spam, the canned grey mail, and dare we say it the false positives. This is the most labour resource intensive part of the testing. I couldn't bear to imagine how many hours/days it would take to go through 22 servers results every week for two or three months.

So the resources like time, budget and labour are against us to complete such a test for this review.

We can however let you in on a few -generic" results derived from the various private anti-spam product testing contracts that we have completed in the past 18 months.

This spam testing experience has shown us that most of the applications we have performed private testing on in an -out of the box" configuration rate at around 65 percent to 70 percent spam catch accuracy with very low to zero false positives. You may well appreciate as with most things in life there are no two e-mail environments which are identical, therefore the anti-spam vendors have a difficult time deciding on default baseline settings to achieve the best spam catch rate with a low to zero false positive rate for the widest possible base of users.

Administrators should expect a certain amount of fine tuning or tweaking to achieve higher catch rates and possibly lower false-positives depending on the industry . Imagine applying an anti-spam filter if you were in the pharmaceutical industry or the porn industry! Both naturally require e-mails to be sent and received containing information about their respective businesses.

A false positive as its name suggests is a legitimate e-mail message, (ham), that a recipient should receive that gets filtered incorrectly and flagged as spam. This is particularly bad if the filter has been set to drop all messages determined to be spam as it means that ultimately the recipient would potentially have no idea that the message was ever sent.

To add another level of complexity to the mix there are also messages which fall into the area between ham and spam these are generally classed as -grey" mail and are the newsletters, circulars etc that some recipients may subscribe to and need to receive. These have many common characteristics of spam messages and are very hard to filter correctly. So at the end of the day there are three bodies of messages to be concerned with, spam, ham and grey. In a perfect world all spam should be dropped, all ham and grey mail should be delivered.

With a few weeks of concerted tweaking and testing by the mail administrator, we managed to increase the spam catch rate to somewhere between 85 percent to 92 percent while still maintaining a zero false positive rate for most of the applications that were tested in private.

Grey mail on the other hand is a very different beast. The best ways we have found to deal with the sensitivity of the filters in respect to these messages is to include them explicitly on a white list.

Newsletters

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.
See All