Saying that it doesn't test client-side vulnerabilities is sort of beside the point. It's a test of malware blocking. And, at least today, vulnerabilities are a much smaller issue for end users than straight-out malware, and especially for IE9. I think the test is a very reasonable one.
>>What about the lack of measurement of vulnerable and outdated browser plugins which could lead to a successful exploitation through a web based malware exploitation kit
But surely these would be blocked, at least in part, by the URL reputation system.
According to a newly released research by NSS Labs, Microsoft’s Internet Explorer 9 greatly outperforms competing browsers in a test against socially-engineered malware. Based on an active testing against 615 malicious URLs for 19 days, both Internet Explorer 9 and Internet Explorer 8 topped the comparative chart.
Here are the findings:
Windows Internet Explorer 9 - IE9 caught an exceptional 92% of the live threats
Windows Internet Explorer 8 - caught 90% of the live threats
Apple Safari 5 - caught 13% of the live threats
Google Chrome 10 - caught 13% of the live threats
Mozilla Firefox 4 - caught 13% of the live threats
Opera 11 - caught 5% of the live threats
More details:
With SmartScreen enabled and Application Reputation disabled, IE9 achieved a unique URL blocking score of 89% and over-time protection rating of 92%. Enabling Application Reputation on top of SmartScreen increased the unique URL block rate of Internet Explorer 9 by 11% (to 100%) at zero hour as well as the over-time protection by 8% (to 100%). Internet Explorer 9 was by far the best at protecting against socially-engineered malware,even before App Rep’s protection is layered on top of SmartScreen.
Why are NSS Labs’ findings not necessarily accurate?
This isn’t the first time I’ve criticized research published by NSS Labs, and definitely not the last. Not only is the research ignoring the existence of client-side vulnerabilities, it’s methodology is fundamentally flawed taking into consideration the limited number of URLs the browsers are tested against, combined with lack of testing of the additional protection features offered by the competing browsers and the related security add-ons.
See:
- Study: IE8’s SmartScreen leads in malware protection
- IE8 outperforms competing browsers in malware protection — again
An excerpt:
By excluding client-side vulnerabilities, the study isn’t assessing IE8’s DEP/NX memory protection, as well as omitting ClickJacking defenses and IE8’s XSS filter, once pointed out as a less sophisticated alternative to the Firefox-friendly NoScript.
Socially engineered malware is not the benchmark for a comprehensive assessment of a browser’s malware block rate. It’s a realistic assessment of the current and emerging threatscape combined with comprehensive testing of all of the browser’s currently available security mechanisms, a testing methodology which I think is not present in the study.
What do you think? Isn’t the fact that client-side vulnerabilities are excluded, undermining the benchmarking methodology used? What about the lack of measurement of vulnerable and outdated browser plugins which could lead to a successful exploitation through a web based malware exploitation kit?
Talkback.
Image courtesy of NSS Labs.
