Debunking yet another bogus malware study

Here we go again, with yet another round of bogus reporting about the extent of malware infections in the United States. This morning I read a report by Nick Farrell of The Inquirer, which was accompanied by the screamer headline One in Four US computers infected. It links in turn to a much longer story in the Sydney Morning Herald, headlined A quarter of US PCs infected with malware: OECD. Here's the lede from that story:

An OECD study into online crime says that increased activity by cyber criminals has left an estimated one-in-four US computers infected with malware.

And a bit later in the story the reporter shows his math:

"It is estimated that 59 million users in the US have spyware or other types of malware on their computers," the OECD report said. According to Nielsen/Netratings, the US internet population stood at an estimated 216 million at the end of 2007.

NewScientistTech (UK) swallowed the story. So did the AFP wire service. In every single one of the press reports I've referenced, the discussion quickly turns to zombies and botnets and Trojans and keyloggers. [Update 3-June: In the credit-where-credit-is-due department, Joel Hruska at Ars Technica deserves props for an excellent report on the OECD study that captures its good work and completely ignores the bogus statistics. I highly recommend reading his post, OECD on malware: it's all about the economics.] [Update 9-June: The OECD has added the following note to the introduction of its report:

Note (9 June 2008): The following sentence p. 37 "Furthermore, it is estimated that 59 million users in the US have spyware or other types of malware on their computers" should read "After hearing descriptions of 'spyware' and 'adware,' 43% of internet users, or about 59 million American adults, say they have had one of these programs on their home computer." The original source can be found in Pew/Internet, "Spyware" July 2005, p.3.

Kudos to the authors of the OECD report for responding to this report and correcting the record.]

The OECD report is wrong to use these numbers, and the reporters who wrote these stories didn't even do any rudimentary fact-checking to see whether the statistics in question were correct. I went back to the original documents and followed the footnotes. This is literally a fourth-hand report from a three-year-old study, and the original research doesn't support anything remotely like the conclusion that's being reported today. It illustrates what is so horribly, horribly wrong with our media in general and our technical press in particular. Here's the real story: In a study conducted three years ago, in 2005, one organization found that roughly 43% of the American computer users they surveyed had experienced at least one go-round with spyware or adware, which they defined as the kind of programs that produce pop-up ads on users' computers. The experience had been so annoying and frustrating for the users they spoke with that 90% of them had changed their behavior dramatically, doing things that would specifically protect them from this sort of infection. From those results, this organization extrapolated that their findings at that time would have equaled 59 million computer users who were being annoyed by adware and spyware programs. So how did we get from that old study to a screaming headline claiming those numbers indicate current infections by malicious software? It took a lot of sloppy work by a lot of people. In this post, I'll break it down for you. Let's start with the report itself. Entitled Malicious Software (Malware): A Security Threat to the Internet Economy (pdf here), it is identified as a Ministerial Background Report for the Organisation for Economic Cooperation and Development (OECD) Ministerial Meeting on the Future of the Internet Economy, to be held in Seoul, South Korea on June 17-18, 2008. The report was produced by the Committee for Information, Computer and Communications Policy of the Directorate for Science, Technology and Industry, which is in turn a subgroup of the OECD. So where does that magical 59 million number come from? You'll find it on page 37, in the middle of a boilerplate section rattling off various statistics from around the world, to illustrate that consumers and businesses "are increasingly exposed to a new range of complex, targeted attacks that use malware to steal their personal and financial information." The quote in question reads as follows:

Furthermore, it is estimated that 59 million users in the US have spyware or other types of malware on their computers. (106)

In the original, that "106" is in superscript, which I can't easily replicate in this post, so I've used parentheses. But anyone who's ever prepared a term paper will recognize that it's a footnote. Let's follow it, shall we? At the bottom of page 37 is this not-so-helpful citation:

106 Brendler, Beau (2007) p. 4.

A perusal of the endnotes finds the full source of this citation:

Brendler, Beau; “Spyware/Malware Impact on Consumers”; APEC-OECD Malware Workshop; April 2007 (Source: StopBadware Project); available online at: http://www.oecd.org/dataoecd/33/55/38652920.pdf (last accessed 13 December 2007).

So, well over a year ago, in April 2007, the group that produced the OECD report invited an American expert to give them a briefing on the extent of malicious software. Page 4 of his PowerPoint presentation includes this sentence:

59 million Americans have spyware or other malicious badware on their computers. (Source: StopBadware Project).

"Badware"? What the hell is that? Well, for starters, it includes a lot more than Trojans, rootkits, and viruses. According to the Stop Badware Project's own definition:

What is badware? There are several commonly recognized terms for specific kinds of badware - spyware, malware, and deceptive adware. Badware is malicious software that tracks your moves online and feeds that information back to shady marketing groups so that they can ambush you with targeted ads. If your every move online is checked by a pop-up ad, it's highly likely that you, like 59 million Americans, have spyware or other malicious badware on your computer.

In fact, software doesn't have to be malicious to be labeled "badware" by the self-appointed sheriffs of the StopBadware Project. Last month, the organization was prepared to apply the label to Apple's Safari browser for Windows. From the StopBadware Project blog:

A few weeks ago, the blogosphere raised concerns about the Windows version of Apple Software Update for offering new software installations (e.g., Safari) disguised as product updates. At the time, we blogged about it and said we were looking into it. It turns out that we were prepared to release an alert today identifying the product as badware. I’m glad to report, however, that we don’t have to, as Apple yesterday released an updated version that addresses the concerns that bloggers and StopBadware.org raised with them.

And how about that 59 million number? It turns out that it doesn't even come from the StopBadware project itself. The source is actually ...

Continue reading on next page -->

<-- Continued from previous page

...a study by the Pew Internet and American Life Project, which was referenced in the original press release announcing the formation of the StopBadware Project in January 2006:

Whether spyware, incessant pop-ups or other obtrusive programs, badware today plagues millions of people by turning their computers into machines to spy on them and steal their data. Unlike viruses and worms, badware becomes embedded in a computer by downloading games or software or just by visiting certain websites. [emphasis added] According to a recent Pew Internet & American Life Project, roughly 59 million American adults today have badware on their computers. Problems related to badware forced home computer users to spend roughly $3.5 billion in 2003 and 2004 to replace or repair their hardware, according to Consumer Reports.

I've bold-faced that one clause to draw attention to it. The organization that originally publicized that "59 million" number specifically excluded viruses and worms! Ironically, when I tried to track down the original Pew Internet report, Google warned me that the site I was trying to get to might not be safe:

And in the most delicious irony of all, when I tried to open the PDF file from the Pew website, Google displayed this message, suggesting that I visit StopBadware.org to learn how to protect myself!

That's ludicrous, of course, but it indicates in hilarious fashion how easy it is to falsely identify a legitimate program or website as a threat. Eventually, I was able to read the Pew Spyware report (pdf format). It was released on July 6, 2005, nearly three years ago. The research was conducted between May 4 and June 7, 2005. And the crucial part of the report, the source of the "59 million" number? Here:

After hearing descriptions of “spyware” and “adware,” 43% of internet users, or about 59 million American adults, say they have had one of these programs on their home computer.

"Have had." Not "have." And in 2005, not 2008. The other key fact in the Pew report is that 91% of the people they surveyed had "changed their online behavior to avoid unwanted software programs." Back in 2004 and 2005, I wrote a lot about spyware and adware. It was a plague at that time. Since then, however, a lot has happened. Windows XP Service Pack 2 dealt a major blow to many of the most common techniques for distributing adware and spyware, and Windows users who were burned by this stuff got a lot smarter about the way they behave, using third-party software and changing the way they interact with the web to protect themselves. Today, adware and spyware are a nuisance, but nowhere near the epidemic levels they were four years ago. I have no idea how many American computer users today are infected with malware (using its generally accepted definition that includes viruses, Trojans, keyloggers, and rootkits). Sadly, the OECD doesn't either, and throwing out alarming and inaccurate statistics like these, which then get amplified by an overworked, undereducated press, doesn't help the cause.