Microsoft fights back on antivirus certification fail, claims malware tests aren't realistic

Microsoft fights back on antivirus certification fail, claims malware tests aren't realistic

Summary: Microsoft took a previous failure to win certification on the chin, but this time it says all antivirus tests struggle with reality.

TOPICS: Security, Microsoft

Microsoft's Security Essentials anti-malware package has again failed to gain approval from German testing firm AV-Test, but Redmond says the malware samples used to assess the software don't reflect real-world conditions.

Security Essentials 4.1 was amongst three of 25 security products tested that failed to gain certification for in AV-Test's November to December tests. Others were Symantec-owned PC-Tools and AhnLab's V3 Internet Security 8.0. The products were tested against malware samples on Windows 7.

While Security Essentials didn't falsely detect malware and blocked all prevalent malware, it missed significantly more zero-day and new malware samples than its rivals, dragging down its performance in the tests.

The industry average for detecting 100 zero-day malware samples used by AV-Test was 92 percent, while Microsoft only achieved 71 percent in November and 78 percent in December. Security Essentials also missed about nine percent of a set of 215,999 malware samples discovered in the past three months. 

Still, it was an improvement on Security Essentials' performance against zero-day threats in October, when AV-Test knocked it off the certified list for the first time in 2012. The last time it failed to be certified was in 2010

Microsoft response

Despite its improved performance in the test, Microsoft malware protection centre programme manager Joe Blackbird challenged the most recent result on the basis that its customers don't encounter the malware samples AV-Test used. 

"When we did our review, we found that our customer-focused processes had already added signatures that protected against four percent of the missed samples. These files affected 0.003 percent of our customers," wrote Blackbird.

Blackbird said Microsoft preferred to measure its performance based on "customer impact" , highlighting the "difficulties and shortfalls" that AV testing organisations have in assessing threats that customers face in the real world.


However AV-Test's CEO Andreas Marx told ZDNet that low prevalence of malware is par for the course today, pointing to the use of "server-side polymorphism" -- a technique designed to evade signature-based defences by slightly altering the malware's appearance without changing its impact, countered by antivirus features such as behavioural analysis.

"Today, every attack is somehow targeted. One example is server-side polymorphism which means that every visitor of a malicious website gets a different variation of the same malware. This means the malware file looks different, but behaves the same. So the prevalence for this sample is very low, as just one user was affected, world wide," Marx said in an email to ZDNet.

Explaining AV-Test's methodology, Marx said it intentionally doesn't test products against millions of samples but rather plucks samples from the major families of malware.

"As of today, every two seconds we see three new malware samples, which are summing up to a few million samples per month. Instead of looking at millions of samples, our focus is on the unique families. Out of every family, we select recent samples in order to use them in our tests. So the impact of these samples is indeed low, however, the impact of the malware family is considerably high. We favor the family-based approach over the sample-based one because of today's malware situation."

Topics: Security, Microsoft

Liam Tung

About Liam Tung

Liam Tung is an Australian business technology journalist living a few too many Swedish miles north of Stockholm for his liking. He gained a bachelors degree in economics and arts (cultural studies) at Sydney's Macquarie University, but hacked (without Norse or malicious code for that matter) his way into a career as an enterprise tech, security and telecommunications journalist with ZDNet Australia. These days Liam is a full time freelance technology journalist who writes for several publications.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • These simulated tests hardly make sense.

    If the German AV-Test firm are so brilliant at anti-virus tech, there should create their own anti-virus software. Creating some silly lab conditions is not going to work in real world scenario.

    No anti-virus software is 100% fool proof, the users themselves have to take other measures to protect their assets.

    Microsoft's Security Essentials is one the best anti virus currently available and in terms of performance, footprint and resource usage its beats every other vendor, at least in my experience.
    • So let me get this straight...

      25 products were tested. 3 failed. One of those three was Microsoft Security Essentials. So the problem must be with silly lab conditions, not Microsoft Security Essentials. Brilliant work, young man. And we wonder why so many IT jobs are being outsourced to India...
      • Just use Linux without AV and don't get infected.

        I've been doing it for over 11 years. You simply won't get infected. Period.
        • So if you...

          afraid to be hit be a car you run from a city to a desert and spent there rest of your life under a rock? I just a insurance and watch my steps...
          • Linux Mint emulates Windows and is so well built it does not need AV.

            Believe me, I would't trust Windows for ANY financial transactions. It gets buried by botnets that are installed without any user intervention. And to top it off, the botnets don't ring any doorbells, so even anti-virus can't detect them. The situation is such people are infected and don't even know it, and start buying online and accessing retirement and investment accounts oblivious that a Russian botnet like Auleron.dx has infected your computer and turned on your proxy setting to a Russian IP address, so the Russian mafia now has the capability to monitor and record all your financial login information.

            Using any Windows is extremely foolish and dangerous. The anonymous shills here using multiple logins here never mention security, Talking about Windows here never addresses it's security, because it is so poorly designed, it's designed to use anti-virus as it's security. Linux has source code so well designed it never needs anti-virus.
        • Android is powered by a Linux kernel

          Yet Android users have to run AV and even with AV, they still get infected.

          Having a Linux kernel doesn't mean the user is safe. Remember, Linux isn't an OS, it is a kernel. All Android users are using Linux and they are not safe.

          PS I've been using Windows even longer than you've been using a Linux kernel powered OS and I haven't ever been hit with malware.
          • But by your statments before

            if the end user installs the malware its not the OS fault its the user..
            You don't have to run AV. I've tried AV on my phone & tablet never found anything..
            Anthony E
          • AVG did

            I have AVG installed on my Nexus 7 tablet and yesterday it popped up during the install of an app telling me that this app was "an adware intrusive app" and that I should uninstall it. I followed its recommendation!

            The only other warning AVG ever gave me was when I manually installed Flash on Android 4.2 and enabled the side-loading by unchecking the "install app from the store only" setting for this. AVG then recommended me to put back the checkbox to install only from the store.
          • Oh

            And Android smartphones are the most infested malware phones out there.
            See and
          • Androids don't get viruses, they get trojans from stupid users.

            If you don't have trusted programs in a trusted repository to install from, anyone is vulnerable to trojans. After all, if you authorize the program to install thinking you are getting some utility and in reality you are also getting a spyware or malware component you don't know about, only you are to blame. Jailbreaking or rooting Android to get apps outside those provided through the Android repository is foolish. But it's exactly the same for any OS.....No, Linux on android does not get viruses, you can take that to the bank. You just don't know the difference between malware and a trojan.
        • You mean like I've been doing with Windows for more than 11 years?

          I can't recall the last time I was infected on my personal PCs. I think it was back in the DOS days.
          • You may want to call Geek Squad, De-bug it and the other repair shops and .

            ...tell them Windows doesn't get infected and they can close up shop and all can go home.

            It's not about you, Ye, people are swimming in problems and infections with Windows. Really, it's the same for any individual here touting how they run Windows and have bee malware free for xxx years.

            The general population can't do it, even with AV and constant scanning. I even got infected with the Auleron.DX botnet just updating my dual boot netbook with Win7 64- bit, I found it and removed it, but any average user would have no indication that all their browser activities were going through a proxy server in Russia since the botnet turned on the proxy setting and set the IP to a server in Russia to connect for the purposes of gleaning financial login information.

            The problem with Windows is that it will become infected with no user intervention. Linux absolutely dose not do that. No one that I know of uses AV with Linux. I have over 10,000 Linux forum emails saved in my Gmail. Not one is complaining about any malware infection or asking how to clear a malware infection. I've never hat do to it in 11 years, even with never having used AV.

            Ye, you have used Linux, Have you ever become infected?
        • I have been using Windows..every version since 3.1 and never been infected.

          SO WHAT? I have seen plenty of infected Linux and Windows machines... most of time because someone opened an email attachment that anyone with any sense wouldn't have.
        • Errrr....

          Figures. A stoned Linux user.
          Who uses a computer without any type of AV/anti-malware protection on any platform is definitely stoned.
          • No, you are just Microsoft brainwashed, and eating it up.

            Microsoft needs the paradigm of AV being necessary because their business mode schleps out cheap, poorly designed insecure code that can be hacked, even when they keep it private and closed source. Anti-virus is Microsoft security for poorly designed source code. Do you realize how much money and manhours Microsoft saves by turning their security to AV companies. (AV companies don't charge Microsoft anything, but provide a lion's share of after-the-fact security to Windows.) You are fooling for using Windows, even with AV. You are literally asking for someone to take your financial login information.
      • Why do you assume all lab conditions are perfect?

        There are plenty of examples where experiments have later been refuted due to poor implementation of the experiment.

        If the test consisted of 5 samples:
        S1: attacked 5 million computers
        S2: attacked 1 computer
        S3: attacked 1 computer
        S4: attacked 1 computer
        S5: attacked 1 computer

        and MSE protected against S1 but none of the others, and all the other AV programs protected against S2-5 but not S1, MSE would "fail" even though it clearly does a MUCH better job of protecting you against malware.

        That is just one example of how this could be a poor test, and it is basically what MS is saying.
        "This means the malware file looks different, but behaves the same. So the prevalence for this sample is very low, as just one user was affected, world wide"
        • depends on how you count

          S2.1 attacked one computer, S2.2 attacked one computer, .... S2.1000000 also attacked one computer. so all together attacked the same one million computers, yet MS doesn't pick this virus up because neither of it's permutations attacked enough computers.
          • Absolutely right

            Your argument doesn't counter the argument that you can't take 100 samples and weigh them all equally. Well, you CAN but you end up with results that aren't representative of the real world.

            "The industry average for detecting 100 zero-day malware samples used by AV-Test was 92 percent"

            If those 100 malware samples aren't weighted, as MS is suggesting, then saying your AV protects against 92% of malware attacks isn't true. If the 70% of the unweighted samples that MSE protects against represents 99.999% of all malware attacks, that shouldn't be a fail.

            You might be right with your example but even if you are, these samples SHOULD be weighted.
          • What does it mean when.

            I have used and installed Linux for a multitude of family, friends and clients for over 11 years and never had even one incident of infection from anyone. Linux does not automatically get infected, even without using AV -- ever.
          • This reminds me of another ridiculous test

            "The industry average for detecting 100 zero-day malware samples used by AV-Test was 92 percent"

            Note the wording here. The 100 samples were chosen by AV-Test. What rationale did they use to pick those 100 samples? By choosing different sets of 100 samples, you can come up with drastically different results, even though nothing has changed in the AV software, or in the real world.

            This reminds me of those html5 tests where the test author chose what unapproved, non standard, proposed html5 features they wanted to include in their test. There was no weighting done based on how likely those individual features were to be accepted in the standard (even if this was possible, which it probably isn't). Browsers that hadn't implemented the non standard features that the test author decided to test against were dinged and determined not to be html5 compliant.

            By choosing different sets of tests (or samples) you can come up with radically different results. In some cases, this is due to poor "lab conditions" (as stated by someone else here). That is MS's claim here. I doubt AV-Test is doing this maliciously, they just used a poor methodology to pick their samples. In the case of the html5 tests, it is clear that the tests were chosen in order to ensure that the author's favorite browser "won" and the author's most hated browser "lost".