Spam free (with a little help from your friends)

Spam free (with a little help from your friends)

Summary: Researchers at UCLA and the University of Florida have created a new type of distributed spam-filtering system that is more efficient and scalable than the alternatives in use today. By using your existing e-mail contact network, this approach can achieve a near-100% spam detection rate with minimal bandwidth cost.

SHARE:
TOPICS: Tech Industry
13

Researchers at UCLA and the University of Florida have created a new type of distributed spam-filtering system that is more efficient and scalable than the alternatives in use today. Results of a large-scale prototype were published in the October edition of IEEE Computer.

The idea is simple:

Spammers send the same or similar messages to thousands of users; we have developed a system that lets users query all of their e-mail clients to determine if another user in the system has already labeled a suspect message as spam.

Social filtering has been tried before (for example, SpamNet), but solutions based on a central server are not scalable. Also, they require building up a totally new social network. This new method uses something you already have - your own personal contact list. A novel "percolation search" algorithm plus a digest-based indexing mechanism minimize network bandwidth and maximize privacy.

To implement the collaborative spam-filtering system, users would first install a plug-in for their e-mail program (though it could also be built-in to future versions of programs like Outlook, or done by large e-mail providers like Hotmail and Yahoo). When a suspect piece of mail arrives, the system uses a random walk of your e-mail contact network to see if someone else has already marked the mail as spam. All messages to other clients are through specially formatted and secure emails. There are safeguards built-in to prevent abuse while at the same time achieving up to a 99.6% spam detection rate in their simulations.

This plug-in doesn't exist yet. It will be some time before implementations are available, and its effectiveness will obviously depend on many people decide to use it. But if widely adopted, the technique has the potential to put a serious dent in the spam problem.

Topic: Tech Industry

Ed Burnette

About Ed Burnette

Ed Burnette is a software industry veteran with more than 25 years of experience as a programmer, author, and speaker. He has written numerous technical articles and books, most recently "Hello, Android: Introducing Google's Mobile Development Platform" from the Pragmatic Programmers.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

13 comments
Log in or register to join the discussion
  • Wow, that was useless information.

    From the article itself, I don't see how this is much different from existing implementations along much the same idea, such as http://pyzor.sourceforge.net/ and http://razor.sourceforge.net/
    mathandmetal
    • Razor and Pyzor

      Razor was the precursor to SpamNet, and Pyzor appears to be something of a clone but the documentation is lacking. Razor and SpamNet share the same network. SpamNet's reliance on central servers limits its scalability according to the article, although they're currently up to about a million users now.

      I was a SpamNet customer for a long time (started during their beta) until I started having trouble using it with newer versions of Outlook. So I dropped it in favor of Outlook's own filters plus 3 levels of spam filtering at the mail gateway. Still, about a dozen or so spams leak through every day and have to be cleaned out by hand. There was leak-thru (false negatives) with SpamNet too. It's very annoying so I'm always glad to see people coming up with new ways to combat it.
      Ed Burnette
      • spamassassin/razor/mozilla

        FWIW, I use spamassassin with the razor plugin and Thunderbird, which seem to catch all but 4 to 5 spam per week here.
        JDThompson
        • Don't know if that's good or not

          That's hard to judge without knowing how widely disseminated your address is and how many spams you get. How about posting your email address in cleartext here and seeing what happens. : )
          Ed Burnette
  • But..But Bill Gates Promised Me Spam Would End Years Ago...

    That's why I still only have 640K in my system, because Bill said so.
    itanalyst
    • 640K

      Wow, you have 640K? I only h
      Ed Burnette
      • Yeah! And IBM...

        ... once said that 5 mainframes would handle all the world's computing needs for the forseeable future.
        bportlock
  • MailFrontier Desktop shines. Now!

    Frustrated with Norton AntiSpam, I spent months researching Anti-Spam products, buying them when necessary, and throwing all of them away, save one: MailFrontier Desktop, now owned by Sonic Wall (http://www.mailfrontier.com/products_matador.html)

    Simply put, the product is really good at identifying spam, and really good at identifying legitimate email. It's easy to install, and to set up. It employs a variety of spam detection methods, including rules based and community based. And with millions of users, it has proven to be scalable.

    You can get a free, fully enabled trial copy at http://www.mailfrontier.com/products_matador.html. After 60 days, you have to pay for an annual subscription, or stop using it.

    After spending many months researching the field, I have kept MailFrontier Desktop and I strongly recommend it. There is a lot of really poor anti-spam software out there, and MailFrontier stands out as a clear winner!

    PS: I have no relation to the company. I am just a happy customer.
    SteveMak
  • NO way your that smart

    i thought microsoft was a god and they promissed the best in security improvemnets.

    >"UCLA and the University of Florida have created a new type of distributed spam-filtering system that is more efficient and scalable than the alternatives in use today."<


    wait till ms finds out, you may be sued for patent infringment, or at least for being smarter than they are.
    not of this world
  • From social filtering to minorities' censorship...

    This is a question; doesn't social filtering moves you toward censoring anything you don't like?

    For example, I and 90% of my friends would tag an email about politics as spam, even if 10% of the rest would consider it a legit email. Here I'm not talking about somebody sending 1 million messages, but some noisy prople who send 100 emails to their friends.


    I think trained bayesian filtering is superior as it does fit my own definition of spam. With social filtering I would need to trust my friends' judgment... and I really don't want to do that.

    However, maybe I'm missing the point, please correct me if you like.

    Regards,

    MV.
    MV_z
    • re: From social filtering to minorities' censorship...

      Perhaps instead of having peers flag messages and spam or not spam, they should assign them arbitrary tags instead. Then you could select what actions to take when messages have certain tags, maybe even treating one friend's tag "X" differently than another's.

      Or maybe the system could automatically judge which friends most share your opinions (on a tag-by-tag basis?) a la stumbleupon.
      johnay
    • More than just your friends

      Well, for one thing the system asks more than just *your* contacts, it asks *their* contacts, and *their* contacts, etc.. (not all of them of course, but a depth-first selection using a random walk and time-to-live). After you get two or three levels away you're getting into people you don't know who probably won't have the same viewpoints.
      Ed Burnette
  • drumming up business

    its the age old method of creating a need for this allegedly revolutionary 'plug in'. capitalize on others desparation. lol fez
    noman5