Microsoft fights back on antivirus certification fail, claims malware tests aren't realistic

Summary:Microsoft took a previous failure to win certification on the chin, but this time it says all antivirus tests struggle with reality.

Microsoft's Security Essentials anti-malware package has again failed to gain approval from German testing firm AV-Test, but Redmond says the malware samples used to assess the software don't reflect real-world conditions.

Security Essentials 4.1 was amongst three of 25 security products tested that failed to gain certification for in AV-Test's November to December tests. Others were Symantec-owned PC-Tools and AhnLab's V3 Internet Security 8.0. The products were tested against malware samples on Windows 7.

While Security Essentials didn't falsely detect malware and blocked all prevalent malware, it missed significantly more zero-day and new malware samples than its rivals, dragging down its performance in the tests.

The industry average for detecting 100 zero-day malware samples used by AV-Test was 92 percent, while Microsoft only achieved 71 percent in November and 78 percent in December. Security Essentials also missed about nine percent of a set of 215,999 malware samples discovered in the past three months. 

Still, it was an improvement on Security Essentials' performance against zero-day threats in October, when AV-Test knocked it off the certified list for the first time in 2012. The last time it failed to be certified was in 2010

Microsoft response

Despite its improved performance in the test, Microsoft malware protection centre programme manager Joe Blackbird challenged the most recent result on the basis that its customers don't encounter the malware samples AV-Test used. 

"When we did our review, we found that our customer-focused processes had already added signatures that protected against four percent of the missed samples. These files affected 0.003 percent of our customers," wrote Blackbird.

Blackbird said Microsoft preferred to measure its performance based on "customer impact" , highlighting the "difficulties and shortfalls" that AV testing organisations have in assessing threats that customers face in the real world.

Methodology

However AV-Test's CEO Andreas Marx told ZDNet that low prevalence of malware is par for the course today, pointing to the use of "server-side polymorphism" -- a technique designed to evade signature-based defences by slightly altering the malware's appearance without changing its impact, countered by antivirus features such as behavioural analysis.

"Today, every attack is somehow targeted. One example is server-side polymorphism which means that every visitor of a malicious website gets a different variation of the same malware. This means the malware file looks different, but behaves the same. So the prevalence for this sample is very low, as just one user was affected, world wide," Marx said in an email to ZDNet.

Explaining AV-Test's methodology, Marx said it intentionally doesn't test products against millions of samples but rather plucks samples from the major families of malware.

"As of today, every two seconds we see three new malware samples, which are summing up to a few million samples per month. Instead of looking at millions of samples, our focus is on the unique families. Out of every family, we select recent samples in order to use them in our tests. So the impact of these samples is indeed low, however, the impact of the malware family is considerably high. We favor the family-based approach over the sample-based one because of today's malware situation."

Topics: Security, Microsoft

About

Liam Tung is an Australian business technology journalist living a few too many Swedish miles north of Stockholm for his liking. He gained a bachelors degree in economics and arts (cultural studies) at Sydney's Macquarie University, but hacked (without Norse or malicious code for that matter) his way into a career as an enterprise tech, s... Full Bio

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Related Stories

The best of ZDNet, delivered

You have been successfully signed up. To sign up for more newsletters or to manage your account, visit the Newsletter Subscription Center.
Subscription failed.