Spam free (with a little help from your friends)

Researchers at UCLA and the University of Florida have created a new type of distributed spam-filtering system that is more efficient and scalable than the alternatives in use today. Results of a large-scale prototype were published in the October edition of IEEE Computer.

The idea is simple:

Spammers send the same or similar messages to thousands of users; we have developed a system that lets users query all of their e-mail clients to determine if another user in the system has already labeled a suspect message as spam.

Social filtering has been tried before (for example, SpamNet), but solutions based on a central server are not scalable. Also, they require building up a totally new social network. This new method uses something you already have - your own personal contact list. A novel "percolation search" algorithm plus a digest-based indexing mechanism minimize network bandwidth and maximize privacy.

To implement the collaborative spam-filtering system, users would first install a plug-in for their e-mail program (though it could also be built-in to future versions of programs like Outlook, or done by large e-mail providers like Hotmail and Yahoo). When a suspect piece of mail arrives, the system uses a random walk of your e-mail contact network to see if someone else has already marked the mail as spam. All messages to other clients are through specially formatted and secure emails. There are safeguards built-in to prevent abuse while at the same time achieving up to a 99.6% spam detection rate in their simulations.

This plug-in doesn't exist yet. It will be some time before implementations are available, and its effectiveness will obviously depend on many people decide to use it. But if widely adopted, the technique has the potential to put a serious dent in the spam problem.