Note to all anti-spam vendors: Why your solutions stink and standards are needed instead

Note to all anti-spam vendors: Why your solutions stink and standards are needed instead

Summary: Whenever I write about spam (a lot recently), the press relations corps for the great many anti-spam solution providers (there must be thousands by now) out there bombard me with e-mails to tell me why the solution they're hocking is that one solution that will blow my mind.  Through the magic of some Kool-Aid they've been drinking (served up by some hyper-excited CTO), they're relatively certain that once I see what it is they've got, I will rave about here in my blog.

SHARE:
158

Whenever I write about spam (a lot recently), the press relations corps for the great many anti-spam solution providers (there must be thousands by now) out there bombard me with e-mails to tell me why the solution they're hocking is that one solution that will blow my mind.  Through the magic of some Kool-Aid they've been drinking (served up by some hyper-excited CTO), they're relatively certain that once I see what it is they've got, I will rave about here in my blog.  Even in the Talkbacks to what I've written about spam, there are countless readers who write about the solution that works for them.  David obviously hasn't seen XYZ.

Guess what. I don't have to see it to know that it's not the answer to the larger spam problem that is choking the Internet, causing users all sorts of grief, and in some cases, either resulting in damages through fraud or malware.

In most cases, I don't bother responding.  There are simply too many people that think they've got it all figured out for me to get back to all of them. So, at best I get back to a few and I ask one very simple question: what does your solution do to guarantee that the e-mail I send to other people doesn't get falsely classified as spam by whatever antispam solution the recipient is using.  This is referred to as spam's "non-deliverability" problem. It's really quite nice that your spam solution has found that perfect balance where it blocks everything that's spam without ever blocking the good e-mail too.  But, as a result of spam, all sorts of solutions are in place -- many of them not nearly as good as yours and as a result, when I send someone and important piece of mail, it's not until I pick up the phone that we figure out that my e-mail never got to them.

Judging by the answers I get, most of the time, the people I send this question to either don't understand what I've asked them, they lie, or they use some easily recognized strain of double-speak that's designed to put me back on the trail of why their solutions do such a great job dealing with inbound spam (when my question had nothing to do with inbound spam).  If you're reading this and saying "he's talking about me," trust me, there are so many e-mails and Talkbacks that go down one of these three paths that it isn't just about you.  But it is most definitely about you and your comrades.

Eventually, the PR corps step out of the way (their brains fried from my insistence that we talk about the "non-deliverablity" problem) so that the CTO at Acme Antispam Company can personally pour me some Kool-Aid at which point, I ask the question again. Eventually, we part ways and the answer ends up being the same from one CTO to the next.  It's quite simple: the recipient needs to be running our solution as well.

In other words, in order for Acme Antispam Company's solution to do as good a job making sure everyone's e-mail safely arrives at their targeted recipients' inboxes as the job does handling inbound spam, everybody (in the world) has to be running it.  The CTO at Acme Antispam Company is usually VERY happy when s/he realizes I've come to this conclusion.  "Yes! Finally!" thinks Acme's CTO "My master plan is nearly complete.  Once David Berlind repeats this secret formula on ZDNet, the world will be mine!!! Ah ha ha ha ha ha ha ha," s/he diabolically laughs. "All mine!!!!!!"

Don't get your hopes up Mr/s. CTO.  First of all, there isn't a snowball's chance in Hell that everyone in the world is going to adopt your solution.  Not only is there just way too much noise from way too many antispam solution providers for any single provider to even come close to this ridiculous goal, neither David Berlind nor ZDNet have that kind of weight with the world.  All this sad, Mr/s. CTO has a great point that I'm constantly reiterating.

It is probably true that if everyone in the world ran just one solution, we'd be able to tweak that solution in such a way that we'd finally get a handle on the inbound and outbound problems associated with spam.  When everyone has access to the same technology, there's a name for that.  It's called a standard.  There is zero chance of some proprietary solution becoming the defacto antispam solution for the world.  But, if only AOL, Google, Microsoft, and Yahoo (the world's leading e-mail solution/service providers)  would get together and decide on what the non-proprietary standards should be and implement them in their systems, it wouldn't be long before every other e-mail solution provider would have to follow suit (in order for their e-mails to interoperate). Pretty soon, guess what? Everyone would have access to the same anti-spam technology and not only would the inbound problem be solved.  So too would the "non-deliverability" problem.

But so long as end-users keep adopting proprietary solutions that do nothing about the deliverability problem (there's not much they can do about what's on the other end of the pipe unless it is their's) and so long as AOL, Google, Microsoft, and Yahoo don't act multilaterally, we'll keep digging ourselves deeper into to the illusion that we've got a handle on the problem when it reality, we're not even close.

Topics: Security, Collaboration

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

158 comments
Log in or register to join the discussion
  • Uh, they are doing exactly that

    In many ways, AOL, Google and Yahoo *are* doing what you ask (and even Microsoft is making encouraging noises).

    The "standard" the industry's heading towards is "true" sender reputation (not the DNS-IP-blacklists-on-drugs that we have today). Being able to store and share opinions about the "goodness" of an individual sender and/or sender domain would be incredibly useful, but we're not there yet -- mainly because email is to easy to forge. This is where sender authentication comes in.

    So the necessary precursor to sender authentication is to get everyone using DKIM, so we have a strong method of sender authentication (not just the relatively weak-but-easy SPF/SenderID) -- this is where the big three mentioned above is right now (and as I said, Microsoft is making encouraging noises, despite its wedded bliss with SenderID).

    For more, see http://richi.co.uk/blog/labels/DKIM.html

    Richi Jennings.
    richij
    • Silly typo

      Sorry I messed up the third paragraph. I meant to write, "The necessary precursor to sender *reputation* is to get everyone using DKIM."

      http://richi.co.uk/blog/2007/10/email-sender-reputation-at-all-david.html
      richij
  • Fixed target

    There is, of course, no reason that a better solution to spam than those currently available would not be proprietary. And if a solution is accepted by all the providers then it will become the standard.

    And then those who send spam will study it closely and find a way to defeat it. There's less chance a spam killer will survive than a DRM scheme.

    But let's say that the inventive, profit-minded people creating spam could be overcome. The impossible happens.

    Spam still won't be solved technologically because the distinction between intended mail and spam is subtle and varies case by case. One easy example: a requested sales presentation sent by email can be exactly the same as a spam mailing. A company handling millions of pieces of email a day is not going to know unless told. And that requires a thoroughgoing, promptly updated recordkeeping arrangement.

    If preventing spam requires a great deal of work on the part of email senders and receivers, and a massive expensive effort by the email carriers, the solution will be considered worse than the disease by many. And if a substantial part of the population or even a single major email carrier opts out, the system is defeated.

    There is only one way to defeat spam, and that's for an insufficient number of people to respond. But here again if people refuse to cooperate, maybe because they like what the spammer sent, the system collapses.

    It's possible to assume that software can do anything demanded. But when those who spam and those who make spamming worthwhile cooperate, the software will be defeated.
    Anton Philidor
    • People are a problem

      Efforts to persuade people not to buy from spammers are laudable.

      Unfortunately, the boring fact is that around 50% of the population are of below-average intelligence.


      Richi Jennings.
      http://richijennings.com
      richij
    • Real spammers are trying to hide

      A verifiable email from a salesman will not be classified as spam because the sender can be verified. The spam that clogs 99.9% of all of our mail boxes comes from people who don't want you to know their real email. So this system will work fairly well.
      Al_nyc
      • "... can be verified."

        The effort on the part of both sender and receiver to verify and approve email may be a reason for senders and receivers to opt out of the effort.

        Also, many of those sending spam are also sellers of lists of email addresses in use. When you opt out of email from one source, you have increased the value of lists to be sold to other spam sources. So those senders do "want you to know their real email."

        Finally, even if sender verification would work "fairly well", that's not enough for Mr. Berlind, who demands a 100% effective system, and noted in one comment that such a system already exists. I think he may have restated his view on whether the perfect system is now available, but he does think (only) open sourcers can find it.

        Because of the practical difficulties we're discussing, I don't believe spam is soluble. I'm not certain it's possible to do significantly better than is being accomplished at present.
        Anton Philidor
        • About as soluble as junk mail

          Let's face facts, junk mail is junk mail doesn't matter if it's electronic or snail mail the problem is the same. In both cases the carriers are willing accomplices to the crime. It's a sad fact that as long as everybody but the target makes money off spam, the vendor paying for the advert, the spammer sending out the spam and the isp's for delivering the spam (getting paid on volume). Basically, only the poor victim is getting screwed. Until we get more of the infrastructure on our side something like say fining the ISP's that send out the spam a thousand bucks per destination and jailing the spammers when we catch them will we get a handle on this issue. Right now the only people motivated to stop the spam are the targets of the spam.
          maldain
    • Review your definitions

      [b]There is, of course, no reason that a better solution to spam than those currently available would not be proprietary. And if a solution is accepted by all the providers then it will become the standard.[/b]

      Proprietary solutions, by definition, [b][i]cannot[/i][/b] be standardized. Proprietary solutions are available from only [b][i]one[/i][/b] provider. If something has become a standard, it is by definition no longer proprietary. If it remains proprietary, it becomes a licensed technology but not a standard.
      JJQ1000
      • de facto standards

        A product used by nearly the entirety of its market is a standard, in fact if not with the imprimatur of an organization.

        And organizations have been known to make standards of solutions from one provider. Sometimes because it's the best available answer to a problem.

        Identifying a standard is not the same as confiscating a product.
        Anton Philidor
  • RE: Note to all anti-spam vendors: Why your solutions stink and standards a

    As an anti-spam vendor - I somehow feel targeted by this.

    I've posted a full response at: http://www.spambutcher.com/art8/587494/ (excerpt below)

    In most scenarios - to stop spam by adoption of new standards, someone has to start rejecting someone else's non-compliant messages. Who goes first?

    I've had a few ideas on how to update email protocols without chaos - but haven't personally taken a lot of time to make it happen. I don't have the energy, resources and possibly smarts to do this in a way that still manages to pay my bills.

    In short - I'm just not good enough. Maybe none of us anti-spam vendors are good enough. So:

    I hereby apologize to David Berlind and the rest of the world on behalf of SpamButcher, and all other anti-spam companies that we haven't somehow completely, and totally eliminated the problem of spam without any risk of false-positives for the price tag of $0.00.

    Sorry. There - happy?
    rich@...
    • You are targeted by this

      The current crop of antispam solutions and their provider have led the world of internet users into a false sense of the problem getting solved. Five years ago, I was told by dozens of antispam vendors how we wouldn't be having this discussion in a few years. Bill Gates was on stage saying that spam would be a thing of the past. Spam has only gotten worse and the non-deliverability problem thanks to antispam solutions has become an achilles heel. It's time for a new approach and I really apologize if this threatens the livelihood of anybody. But there are just certain areas of interop where open standards are required and reliable email is one of them.

      Regarding the fact that someone has to stop rejecting mail...I agree 100 percent. After settling one whatever standards (be it DKIM, SPF, or whatever) that they will all comply with, AOL, Google, Microsoft and Yahoo should all agree to a launch date (one that gives the rest of the world plenty of time to catch up) at which point such rejections will start across all of their systems. They need to launch a big publicity campaign so that no one misses the news and trust me, the entire system will come in line VERY quickly since nobody wants their solution or service to be the odd man out.

      It's that simple.

      db
      dberlind
      • Big Publicity Campaign

        That sounds [b]GREAT[/b]! And we can send out a mass email to let everyone know about it. Maybe we can get some of the spammers to help us with that email campaign!?!?!
        sstew9@...
      • Why hasn't David Berlind fixed the spam problem?

        > The current crop of antispam solutions and their provider have led the world of internet users into a false sense of the problem getting solved.

        Sorry if some of my contemporaries are lying in their marketing.

        I see radical claims made periodically and it annoys me. Doesn't mean that all anti-spammers are worthy of your wrath.

        SpamButcher and other similar products are like cold medicine for colds. Or - perhaps more topically - like valium for anger management.

        We don't cure the problem - but for some people we make the symptoms tolerable. I won't pretend there aren't sometimes side-effects.

        > After settling one whatever standards (be it DKIM, SPF, or whatever)

        It would need to be "whatever" as DKIM and SPF don't actually solve the spam problem.

        >AOL, Google, Microsoft and Yahoo should all agree to a launch date (one that gives the rest of the world plenty of time to catch up) at which point such rejections will start across all of their systems.

        Are you aware of our country's anti-trust laws? Even if this could be negotiated between the major ISPs - the minor ISPs would have a fit - and quickly involve the FTC.

        Ain't gonna happen.

        What annoys me is that you're blaming ---me--- (and my competitors) for these problems.

        This is like getting angry at Toyota for only trying to make more efficient cars - not lobbying for more mass-transit.

        It's not like SpamButcher and other anti-spam vendors are trying to prevent change (or even have the ability to). I like having my cash cow - but AOL / MS / Yahoo dwarf the anti-spam companies. They can do what they want.

        In fact - I suspect you have a lot more influence on AOL and company than a small anti-spam player like myself does.

        I hereby am shifting the blame back to you for not solving this.

        Why haven't you fixed the spam problem David? I am so tired of you not fixing the spam problem. You complain and complain about the spam problem - why haven't you fixed it?

        Isn't every else here tired of David Berlind not having fixed the spam problem?

        http://www.spambutcher.com/art8/587494/
        rich@...
        • spam vendors dont want to fix the problem

          As it would be akin to virus companies fixing the virus problem.... the equivalant of shooting themselves in the head
          waylander
          • no - us spam vendors -can'-t fix the problem

            I'm sorry - but this is like saying trauma surgeons don't want people not to get in car crashes.

            Those evil car-crash-loving trauma surgeons! We should be outraged!

            But OK - you caught me.

            I know how to solve the spam problem - but I'm not telling anyone about it:

            http://www.spambutcher.com/press9/698390/
            http://www.spambutcher.com/press9/233344/

            And - us anti-spam vendors do everything we can to make sure users don't have access to free information or services that would help minimize their spam problem:

            http://www.spambutcher.com/spamfreeze/
            http://www.spambutcher.com/art1/274152/
            rich@...
  • RE: Note to all anti-spam vendors: Why your solutions stink and standards a

    SMTP works, if you actually use the tools available to get rid of the noise.

    By being just a little pedantic (aka running greylisting), you get rid of somewhere in the range 85-95% of the spam. The rest you feed to the content filtering such as spamassasin and clamav.

    The made up addresses in your own domain which you inevitably harvest from backscatter, you can put on a bait list and you send anybody who tries to send to those to the tarpit. That rids you of most of the noise that somehow got around greylisting.

    My too short or too long paper (depending on your perspective) about this (for BSDCan last) you can find at <http://home.nuug.no/~peter/malware-talk/silent-network.pdf>, blog posts about some of the spammer baiting you can find at <http://bsdly.blogspot.com/> - just browse, you'll find refs too.
    pitrh
    • Even though David...

      didn't like my previous response, I still contend that verifying MX records and A records (Postfix does an excellent job) I've cut down nearly 90% of the spam coming into my system without having to run spam filters.

      Properly configured mail servers verifying senders through MX and A records will stop spam and allow mail from other properly configured servers to pass.

      Berlind is asking for something new because I don't believe he understands how SMTP works. And if admin's wont use what we already have in place, they aren't going to use something new.
      bjbrock
      • I am not asking for anything new

        First, I understand how SMTP works. Second, I'm not poo-pooing your approach. What I AM saying is that AOL, Google, Microsoft, and Yahoo must all adopt some approach and tell the world that on such and such a date, they will begin to reject e-mails that don't comply. Deliverability is a major problem. What you are suggesting is to follow your approach as a standard approach in lieu of using proprietary antispam solutions all of which have problems with deliverability of legitimate e-mail. If your approach works, I don't need the antispam solution and then the deliverability problem goes away. In other words, to fix the deliverability problem, everyone needs to dump their antispam solutions. I agree 100 percent. I care much less about the actual solution than I used to. The world just needs to pick one and go with it. Until GAMY announces that it will reject e-mails that don't conform with whatever they pick multilaterally, don't expect people to dump those proprietary solutions anytime soon in which case, the deliverability problem (which is worse than the spam itself if you ask me) persists. What good is that?

        db
        dberlind
        • The reason these majors don't...

          force DNS checks is because many small legitimate mail servers have incorrect DNS setup. Some don't even use MX records. And they are sending from a NAT'ed PC which means no A record. I think it would be better to educate these admin's that they are not following RFC's and to fix their issues. Then DNS checks will work and anti-spam software, which is flaky, can be done away with.
          bjbrock