Microsoft, IE security, NSS and the vendor funded study conundrum
For NSS Labs president Rick Moy the questions about his company's study of Internet Explorer security must get old. It's a bit like the movie Groundhog Day. Microsoft finds IE is more secure than other browsers according to a study by NSS and then Moy gets questions about the data and methodology because the software giant funded the research.
Every time there's a NSS study, Microsoft markets the results. However, the NSS relationship started as an engineering relationship. Microsoft found NSS' backed up its own findings and it's pitch city.
Today's headline: IE 9 was able to stop 99 percent of socially engineered malware. IE 8 checks in with 90 percent. The rest of the browsers don't come close. I'm so skeptical about vendor funded studies that I usually don't bother with the briefings. For some reason---slow news day and perhaps curiosity---I took this one. The headlines are all fairly predictable: Microsoft funded study claims IE most secure.
Here are the results based on NSS' research.
Those findings were based on 636 validated malicious URLs. Specifically speaking:
From an initial list of 8,000 new suspicious sites, 1,209 potentially-malicious URLs were pre-screened for inclusion in the test and were available at the time of entry into the test. These were successfully accessed by the browsers in at least one run. We removed samples that did not pass our validation criteria, including those containing adware or that were not valid malware. Ultimately 636 URLs passed our post-validation process and are included in the final results, providing a margin of error of 3.88% with a confidence interval of 95%.
Moy says the NSS methodology and findings are solid. NSS validated the URLs by sandboxing them and watching the malware work. The methodology is the same one NSS does for anti-virus testing and enterprise studies.
However, there's still skepticism. I asked---and should receive---the spreadsheet of all the malware URLs. Moy's point is that this study is picked over more than the rest because Microsoft paid for it. From the NSS perspective, it has to monetize the research somehow. NSS started in 1991 doing testing for tech magazines. From there it branched out with enterprise consulting.
In my interview with Moy, it was clear that the methodology questions were getting old. Microsoft was doing briefings and Moy was getting pelted with questions. His key points:
- NSS does its own URL sourcing.
- Moy has invited all of the browser vendors to participate.
- He doesn't "want to be too nice to Microsoft," but the results make some sense. Microsoft for years was a security doormat. It has improved and put a lot of resources behind Internet security. And Microsoft has the installed base to develop a built in blacklist reputation service rivaling antivirus providers. "Why would we think a small company like Opera would do better?" asked Moy.
- NSS stands by its data despite its relationship with Microsoft. "We see where the question comes up," said Moy. "We stand by the data. This is where the perception issue comes up, but the data is the data." He added that if NSS did the study for any other vendor the questions wouldn't be nearly as numerous. "This was funded, but it's hard to get this data otherwise," said Moy. "It's a hard test to run and no one is set up to do it."