Debunking yet another bogus malware study

Debunking yet another bogus malware study

Summary: Here we go again, with yet another round of bogus reporting about the extent of malware infections in the United States. A wide range of news agencies are reporting the results of a new report that supposedly reveals that one in four computers in the U.S. are infected with malware. But all it takes is a little digging to discover that this is literally a fourth-hand report of data originally gathered in 2005. The most ironic part? When I tried to view the original report from the Pew Internet project, Google stopped me with a false warning that I was about to visit an unsafe site.

SHARE:
151
Here we go again, with yet another round of bogus reporting about the extent of malware infections in the United States. This morning I read a report by Nick Farrell of The Inquirer, which was accompanied by the screamer headline One in Four US computers infected. It links in turn to a much longer story in the Sydney Morning Herald, headlined A quarter of US PCs infected with malware: OECD. Here's the lede from that story:
An OECD study into online crime says that increased activity by cyber criminals has left an estimated one-in-four US computers infected with malware.
And a bit later in the story the reporter shows his math:
"It is estimated that 59 million users in the US have spyware or other types of malware on their computers," the OECD report said. According to Nielsen/Netratings, the US internet population stood at an estimated 216 million at the end of 2007.
NewScientistTech (UK) swallowed the story. So did the AFP wire service. In every single one of the press reports I've referenced, the discussion quickly turns to zombies and botnets and Trojans and keyloggers. [Update 3-June: In the credit-where-credit-is-due department, Joel Hruska at Ars Technica deserves props for an excellent report on the OECD study that captures its good work and completely ignores the bogus statistics. I highly recommend reading his post, OECD on malware: it's all about the economics.] [Update 9-June: The OECD has added the following note to the introduction of its report:
Note (9 June 2008): The following sentence p. 37 "Furthermore, it is estimated that 59 million users in the US have spyware or other types of malware on their computers" should read "After hearing descriptions of 'spyware' and 'adware,' 43% of internet users, or about 59 million American adults, say they have had one of these programs on their home computer." The original source can be found in Pew/Internet, "Spyware" July 2005, p.3.

Kudos to the authors of the OECD report for responding to this report and correcting the record.]

The OECD report is wrong to use these numbers, and the reporters who wrote these stories didn't even do any rudimentary fact-checking to see whether the statistics in question were correct. I went back to the original documents and followed the footnotes. This is literally a fourth-hand report from a three-year-old study, and the original research doesn't support anything remotely like the conclusion that's being reported today. It illustrates what is so horribly, horribly wrong with our media in general and our technical press in particular. Here's the real story: In a study conducted three years ago, in 2005, one organization found that roughly 43% of the American computer users they surveyed had experienced at least one go-round with spyware or adware, which they defined as the kind of programs that produce pop-up ads on users' computers. The experience had been so annoying and frustrating for the users they spoke with that 90% of them had changed their behavior dramatically, doing things that would specifically protect them from this sort of infection. From those results, this organization extrapolated that their findings at that time would have equaled 59 million computer users who were being annoyed by adware and spyware programs. So how did we get from that old study to a screaming headline claiming those numbers indicate current infections by malicious software? It took a lot of sloppy work by a lot of people. In this post, I'll break it down for you. Let's start with the report itself. Entitled Malicious Software (Malware): A Security Threat to the Internet Economy (pdf here), it is identified as a Ministerial Background Report for the Organisation for Economic Cooperation and Development (OECD) Ministerial Meeting on the Future of the Internet Economy, to be held in Seoul, South Korea on June 17-18, 2008. The report was produced by the Committee for Information, Computer and Communications Policy of the Directorate for Science, Technology and Industry, which is in turn a subgroup of the OECD. So where does that magical 59 million number come from? You'll find it on page 37, in the middle of a boilerplate section rattling off various statistics from around the world, to illustrate that consumers and businesses "are increasingly exposed to a new range of complex, targeted attacks that use malware to steal their personal and financial information." The quote in question reads as follows:
Furthermore, it is estimated that 59 million users in the US have spyware or other types of malware on their computers. (106)
In the original, that "106" is in superscript, which I can't easily replicate in this post, so I've used parentheses. But anyone who's ever prepared a term paper will recognize that it's a footnote. Let's follow it, shall we? At the bottom of page 37 is this not-so-helpful citation:
106 Brendler, Beau (2007) p. 4.
A perusal of the endnotes finds the full source of this citation:
Brendler, Beau; “Spyware/Malware Impact on Consumers”; APEC-OECD Malware Workshop; April 2007 (Source: StopBadware Project); available online at: http://www.oecd.org/dataoecd/33/55/38652920.pdf (last accessed 13 December 2007).
So, well over a year ago, in April 2007, the group that produced the OECD report invited an American expert to give them a briefing on the extent of malicious software. Page 4 of his PowerPoint presentation includes this sentence:
59 million Americans have spyware or other malicious badware on their computers. (Source: StopBadware Project).
"Badware"? What the hell is that? Well, for starters, it includes a lot more than Trojans, rootkits, and viruses. According to the Stop Badware Project's own definition:
What is badware? There are several commonly recognized terms for specific kinds of badware - spyware, malware, and deceptive adware. Badware is malicious software that tracks your moves online and feeds that information back to shady marketing groups so that they can ambush you with targeted ads. If your every move online is checked by a pop-up ad, it's highly likely that you, like 59 million Americans, have spyware or other malicious badware on your computer.
In fact, software doesn't have to be malicious to be labeled "badware" by the self-appointed sheriffs of the StopBadware Project. Last month, the organization was prepared to apply the label to Apple's Safari browser for Windows. From the StopBadware Project blog:
A few weeks ago, the blogosphere raised concerns about the Windows version of Apple Software Update for offering new software installations (e.g., Safari) disguised as product updates. At the time, we blogged about it and said we were looking into it. It turns out that we were prepared to release an alert today identifying the product as badware. I’m glad to report, however, that we don’t have to, as Apple yesterday released an updated version that addresses the concerns that bloggers and StopBadware.org raised with them.
And how about that 59 million number? It turns out that it doesn't even come from the StopBadware project itself. The source is actually ...

Continue reading on next page -->

<-- Continued from previous page

...a study by the Pew Internet and American Life Project, which was referenced in the original press release announcing the formation of the StopBadware Project in January 2006:

Whether spyware, incessant pop-ups or other obtrusive programs, badware today plagues millions of people by turning their computers into machines to spy on them and steal their data. Unlike viruses and worms, badware becomes embedded in a computer by downloading games or software or just by visiting certain websites. [emphasis added] According to a recent Pew Internet & American Life Project, roughly 59 million American adults today have badware on their computers. Problems related to badware forced home computer users to spend roughly $3.5 billion in 2003 and 2004 to replace or repair their hardware, according to Consumer Reports.
I've bold-faced that one clause to draw attention to it. The organization that originally publicized that "59 million" number specifically excluded viruses and worms! Ironically, when I tried to track down the original Pew Internet report, Google warned me that the site I was trying to get to might not be safe: Google declares spyware study unsafe And in the most delicious irony of all, when I tried to open the PDF file from the Pew website, Google displayed this message, suggesting that I visit StopBadware.org to learn how to protect myself! Google declares spyware study unsafe (2) That's ludicrous, of course, but it indicates in hilarious fashion how easy it is to falsely identify a legitimate program or website as a threat. Eventually, I was able to read the Pew Spyware report (pdf format). It was released on July 6, 2005, nearly three years ago. The research was conducted between May 4 and June 7, 2005. And the crucial part of the report, the source of the "59 million" number? Here:
After hearing descriptions of “spyware” and “adware,” 43% of internet users, or about 59 million American adults, say they have had one of these programs on their home computer.
"Have had." Not "have." And in 2005, not 2008. The other key fact in the Pew report is that 91% of the people they surveyed had "changed their online behavior to avoid unwanted software programs." Back in 2004 and 2005, I wrote a lot about spyware and adware. It was a plague at that time. Since then, however, a lot has happened. Windows XP Service Pack 2 dealt a major blow to many of the most common techniques for distributing adware and spyware, and Windows users who were burned by this stuff got a lot smarter about the way they behave, using third-party software and changing the way they interact with the web to protect themselves. Today, adware and spyware are a nuisance, but nowhere near the epidemic levels they were four years ago. I have no idea how many American computer users today are infected with malware (using its generally accepted definition that includes viruses, Trojans, keyloggers, and rootkits). Sadly, the OECD doesn't either, and throwing out alarming and inaccurate statistics like these, which then get amplified by an overworked, undereducated press, doesn't help the cause.

Topics: CXO, Hardware, Malware, Security

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

151 comments
Log in or register to join the discussion
  • Excellent analysis

    I guess that people have come to expect bad news so much that journalists no longer verify their sources; if it's bad news about computer threats, it must be true.

    To be fair, this sort of problem (relying on bad sources that themselves rely on bad sources) is a problem everywhere, even in academia. I remember one instance where there was this footnote that relied on an old source, which relied on an even older source and so forth, ending in a bogus claim dating from the early 19th Century.
    LB75
    • THANK GOD FOR A GOOD REPORTER!

      For the longest time I've seen journalism on every front become a dying art.

      The simplest functions of journalism is to CHECK THE FACTS and report a story.

      Lately it's been a STORY,then a pursuit of facts to back up the story - leaving half baked sources or no sources at all and leaving out most of the relevant facts!
      JABBER_WOLF
      • RE: Debunking yet another bogus malware study

        The absolute <a rel="follow" href="http://las-vilis.com/roulette/">online casino roulette</a> designed for <a rel="follow" href="http://las-vilis.com/videopoker/">videopoker online</a> age is en route for accept <a rel="follow" href="http://las-vilis.com/roulette/">it</a> as an another.
        online slots
  • Ed; You're Not Doing Your Part In Propagating The Paranoid!

    Dear Ed;

    I read a good many articles that you and Adrian write here. However, I cannot tolerate the (lack of) fiction in your piece today.

    How do you expect the Anti-Malware lobbyists to finance this company if you are writing about the reality of infected computers and personal PCs in America?

    No sir, you have come against PCTools, Symantec, and a whole host of others in publishing this article. You will pay for this dearly!

    In America we thrive on fear and mega-amounts of FUD to get us through each day, how dare you sir!
    Kromaethius
    • Good point....

      Reading through the posts in this thread, i keep wondering why there's sometimes such a lust for bad news, despite the facts that tell another tale.
      cgdams
      • Thanks, I was just being funny...

        I was just being sarcastically funny when I wrote my original post. However, what you say is true enough and the Media, the fourth branch of the U.S. Government does pride itself on sensationalism and fear. Also I like to add, sex, sex always sells and never lets the media down.

        Scandals, rumors of covertness, paranoia, strife, and violence is also the mainstay of the media.
        Kromaethius
      • RE: Debunking yet another bogus malware study

        Absolutely agree with your opinion on <a href="http://www.shoppharmacycounter.com/">weight loss pills</a>.
        cool_weightlosspills
  • Just another example of the circular "proof" used...

    ...to attack Windows.

    What I find the most interesting is the part where the users stated they had changed their behavior. This implies they were the weak link.
    ye
  • RE: Debunking yet another bogus malware study

    I would suggest that you put the entire OECD report in context, and instead of picking nits on the statistics they produce (I generate tons myself, as an ISP postmaster with 40 million ++ users, and I know statistics can vary a lot - and I also know just how capable media is of misinterpreting them).

    The solutions OECD and APECTEL have put forward are actually excellent and practical. Developed over a period of about four years and hundreds of hours of work with other organizations in this space .. regulators, CERTs, ISPs, individual tech experts

    Having spoken at some of the OECD-APECTEL conferences on malware over the past few years .. conferences where work was done that led to this report], I can confidently say that they have met with and discussed this issue with some of the best people in this space that I can think of.

    srs
    http://www.hserus.net
    hserus
    • Not nitpicking

      How many people read this whole report? Hundreds? Maybe a few thousand. How many read the news reports based on it? Hundreds of thousands at least, maybe millions.

      If their goal is to educate the public and change policy among governments worldwide, then OECD has a responsibility to reach out to the media with accurate information.

      They could have issued an easy-to-read press release highlighting the key recommendations in their report. But instead we get this crap, and the reason is because these terrible, inflammatory, inaccurate statistics were included in the report. Including those statistics was a mistake. They weren't necessary, but there they are in the report, and now on the news.
      Ed Bott
      • Did you read the title of that report, Ed?

        That wasnt a public education document at all.

        It was a ministerial briefing. A high level policy document, with fairly broad and high level recommendations - meant for government ministries in the OECD economies (that'd be the USA, Canada, various European countries, Japan, Korea, Australia..).

        That kind of document (I know for certain) takes upto three to four years of work among several agencies + law enforcement + industry + cert etc groups - see all the other cites there, not just Beau @ Consumer Reports'

        In most cases, the people, and their organizations cited in the report, actually worked jointly on this over the last few years, producing quite a lot of very useful work, international cooperation across government / industry / NGO groups ...

        So, work is being done. And if you want to rail at clueless reporting - you might want to rail at the reporters, rather than at the OECD.

        Having done my share of journalism several years ago, and having been quoted in several stories on spam over the years, I've run into clued reporters, and I've also run into people with less functioning brain cells than the average amoeba. Pity, but I suppose that's sturgeons law (99% of everything is cr*p).

        srs
        hserus
        • Not only did I read the title...

          I cited it in my post here.

          They made the document public. They issued a press release announcing the conference. They invited the media to attend. They have a press page proudly listing links to all the stories I mentioned here.

          Most important of all, they published the erroneous "facts" in this report.

          So please don't tell me OECD is innocent.

          Btw, you got Sturgeon's Law wrong: It's actually "Ninety percent of everything is crud."
          Ed Bott
          • Sturgeon's Revelation

            Actually, it's "Sturgeon's Revelation" - remember a piece by Sturgeon himself, in which he pointed out that "Sturgeon's Law" was something else entirely.

            And it's "crap", not "crud".

            And i was present when the late Robert Bloch promulgated Bloch's Corollary to Sturgeon's Revelation: "...and your agent gets the other ten percent."
            fairportfan
          • Original was "crud"

            From Wikipedia (and corroborated by other authoritative sources):

            In 1951, Sturgeon coined what is now known as Sturgeon's Law: "Ninety percent of SF [science fiction] is crud, but then, ninety percent of everything is crud." This was originally known as Sturgeon's Revelation; Sturgeon has said that "Sturgeon's Law" was originally "Nothing is always absolutely so." However, the former phrase is now widely referred to as Sturgeon's Law.
            Ed Bott
          • Damn!

            Sturgeon beat me to the same sort of idea.

            ?At least 80% of what people post here is absolute nonsense.
            80% of the remaining may be interesting, but still useless.
            80% of what's left after that may be of some use, but will never be acted on.
            Only 1% of the total posting will ever have something that people are likely to ever use.
            I call this the Zinj Pareto-Cubed Law.?
            Dr_Zinj
          • Leaving aside this debate on numbers ..

            .. and the fact that publication of these reports, and their release to media in press release form, is routine ..

            I, for one, do think that they're on the money, on the problems they cite, and the methods of mitigation that they suggest.

            Most of these stats are inevitably going to be back of the envelope estimates. But these can be surprisingly accurate.

            Analysis below from my good friend Joe St.Sauver of UOregon (who I would count as one of the very few people who know what they're talking about, in this space) -

            --------

            Assume, for example that there are a couple of hundred active spammers (this is consistent with Spamhaus' estimates, for example).

            Assume that each spammer wants to spread their spam out over something between 10 and thousand fresh botted hosts per day, using new (unblocklisted) hosts every day, day-in-day out, 365 days a year.

            That would imply something between 200*10*365 and 200*1000*365 infected hosts per year, ignoring things like reinfection, dynamic host effects, etc. That's 730,000 to 73,000,000 botted hosts/year. But hey, some
            spam isn't sent from bots, some spammers are slackers, reinfection does occur, etc., so let's knock that back 20% or so...

            The upper bound would then be 0.8 * 73,000,000 ==> 58,400,000 (and given that there are 5.2 million hosts on the CBL right now, with that listing turning over virtually 100% over a period of weeks if not days), I don't think 58.4 million hosts/year is
            absurd...

            And in fact, that's a number I showed way back in March 2005 in http://www.uoregon.edu/~joe/zombies.pdf

            For context, IDC said that 152 million new PCs were shipped worldwide in 2003. Let's assume that sales have been flat over time, so that an equivalent number of systems were sold last year, and the year before that, etc.

            Assuming a four year PC lifecycle, that implies roughly 600 million PCs in circulation. Less than 10% of that total would need to be owned to meet the 58,400,000 number.

            I find a 10% infection level to be *quite* plausible, to tell you the truth...

            Or look at it from another point of view:

            http://www.internetworldstats.com/stats.htm says that there are 1,407,724,920 Internet users worldwide. Let's assume that half those people have a computer (in reality, many have more than one, and maybe 2/3rds have zero -- but I think that balances out).

            Half of 1,407,724,920 is 703,862,460 computers. If a little over 8.5% of that number of computers was compromised, you'd be at the 59 million mark...

            ---------
            hserus
    • But then doesn't OECD have an agenda?

      Nothing sells like fear, and the "worse" an entity can make an issue sound, the more likely someone listening to them will continue to "need" that entity's help.

      And usually at a cost...
      AllKnowingAllSeeing
  • Sorry to say...

    My personal experience suggests that the numbers are close to accurate. Just about every time I'm asked to diagnose problems on a Windows machine over the past several years, I've found a spyware/adware infestation (usually a severe one).

    BTW: About the only good thing I can say about Windows Vista, is that the vast majority of the machines I've seen with spyware infestations run Windows XP (haven't seen any on Vista).
    John L. Ries
    • The security experts I've spoken with disagree with you

      They say current infection rates for all sorts of malware are roughly 10%, with the really bad stuff (Trojans, bots, etc.) being in the fraction of a percent range.

      Remember that your sample is based on people bringing ill-performing machines to you. In other words, they're self-selecting.
      Ed Bott
      • Hehe, nice one

        [i]Remember that your sample is based on people bringing ill-performing machines to you. In other words, they're self-selecting.[/i]

        I immediately thought of the exact same thing when I read his post. I wonder how many people bring him their machine and say "Everything is working fine, just wanted to bring it in for you to admire how nice my desktop is." :)
        NonZealot