Antivirus tests need better methodology

Antivirus tests need better methodology

Summary: While most antivirus tests do not have the right assessment methods to determine their effectiveness, vendors cannot afford to ignore them as they are key to branding.

SHARE:

Antivirus assessment studies may lack the right methodology to determine how effective antivirus products are, but vendors still need to take them seriously as they are key to branding.

According to Simon Piff, associate vice president of enterprise infrastructure and research at IDC Asia-Pacific, an effective antivirus is able to catch all types of malware, not just a sample of malware.

However, the test's selection process will always affect the results since it is no longer "all the malware in the universe", but a "representative sample", which is unable to fully reflect the universe of malware accurately, he explained.

Most tests too, do not have an accurate methodology in measuring the effectiveness of antiviruses, Peter Stelzhammer, vice chairman of testing firm AV-Comparatives, added. The most accurate test of antivirus efficiency is passing it through a real-life scenario with a statistically valid number of test cases, he noted.

Work on malware detection, don't be intrusive to users

Aside from scoring well in antivirus tests, vendors should also continue innovating and improving their products to detect malware better, because malware evolves at a fast rate and millions of new viruses are introduced everyday, Piff noted.

The most important thing an antivirus should do is protect a user without having too much impact to the system's performance or "annoying" users with too many pop-ups, messages or questions.

In response to a call for comments, Twiiter user @gmanka said he wanted "something that actually protects your PC and isn't intrusive while it's at that". Another consumer @asian_angel said she wanted something easy to set up and use, reliable and did not eat up a lot of system resources.

Singaporean consumer Liang Wen Min however, said she did not mind her software being intrusive as long as it did its job.

She explained her antivirus had been "useless" despite not compromising her PC's performance and left her systems infected with ransomware.

Antivirus tests stir controversy
Their comments come after effectiveness tests conducted on antivirus products in December stirred controversy among security vendors.

Security vendor Imperva released a study in December, where it used an online tool VirusTotal to pit antivirus products against 82 randomly collected malware samples, examining how successful they were in detecting them. It was found from the study anti-malware software was not fast or responsive enough to combat targeted threats, and security software is better at detecting malware that spreads rapidly in massive quantities of identical samples.

German testing firm AV-Test too, put 25 antivirus products for their November to December 2012 study, and found only 92 percent of the zero-day attacks were blocked during the test. The products were able to clean 91 percent of the infected systems, but only 60 percent could be put back in a condition similar to the pre-infection state, the firm said.

Antivirus vendors have retaliated against these studies, attacking the test methodologies. Microsoft for one, told ZDNet last week the use of malware in tests did not reflect real-life conditions, and their customers did not encounter the malware samples used. Microsoft's Security Essentials had only achieved 71 percent in November and 78 percent in December for AV-test's study.

Responding to the Imperva study, Kaspersky Labs also told PC Mag, the VirusTotal service did not use the full versions of antivirus products, but relies merely on a standalone scanner. "This approach means the majority of protection technologies available in modern antivirus software are ignored. This also affects proactive technologies designed to detect new, unknown threats," Kaspersky Labs said.

Explaining the test methodology in an earlier report, Andreas Marx, AV-test's CEO said the focus of the test had been on samples from major malware families. Even though getting hit by these samples are low, getting hit by the families of these samples are high, and an antivirus that is able them has demonstrated effectiveness

Important for branding
That said, antivirus tests are not completely irrelevant as it can prove a good branding opportunity, Stelzhammer pointed out.

When antivirus vendors score well across a variety of tests, they prove their quality and performance, which is way more effective than claiming to have the best product on the market, he explained.

BitDefender for example, was named product of the year in AV-Comparatives' test in December last year. "We feel that it improves consumer perception of our antivirus and moving forward, we will strive to ace more antivirus tests for better branding," the antivirus vendor said.

Topics: Security, Software Development

Ellyne Phneah

About Ellyne Phneah

Elly grew up on the adrenaline of crime fiction and it spurred her interest in cybercrime, privacy and the terror on the dark side of IT. At ZDNet Asia, she has made it her mission to warn readers of upcoming security threats, while also covering other tech issues.

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

6 comments
Log in or register to join the discussion
  • Antivirus

    Yup!! That's correct, anyways customer's reviews also matters, my vote is to comodo antivirus and it's internet security prroducts, for details,
    Visit: http://www.comodo.com/
    Manov Rao
  • Antivirus Methodology

    The problem in the industry is that there is no industry wide definition about what a virus is.

    What some vendors call 'virus', other call 'malware' or 'spyware' or, some other term which has its own meaning.

    Virus' infect other files, Malware uses up system resources, spyware sends out personal info or business documents, and so on.

    Another problem is why would one anti-virus vendor share details about a new exploit/vulnerability with another vendor, and by doing to they are actually beefing up the competitions product? This makes no business sense to do so.

    We definitely need better security products and standards and more user education.

    EgoDust
    EgoDust
    • RE:why would one anti-virus vendor share details about a new exploit/vulner

      This is good questions, in my opinion every vendor who signup to Microsoft MAPP should do it.
      Mr.SV
  • What one should know about antivirus test

    The first place tells you who sponsored it
    nitekatt
  • Humm . . .

    Does Impervia sell products?

    Apparently so. They seem to be "call us if you want to discuss pricing" style products (why anybody buys something that doesn't list the price, I'll never understand), but yes they sell products.

    So - they do have a vested interest in claiming their products are better than AV.

    And "AV" is a misnomer anyways - even the AV vendors recognize things are changing, and rightly point out that their products are rarely simple signature-based scanners anymore.
    CobraA1
  • Best Antivirus of 2013

    Antivirus Test 2013 - Who offers the best antivirus program in the test?

    Online Antivirus Test 2013 - Which is the best antivirus program in the test? We tested eight different programs for you. Buy the Test!

    1 Place: Trend Micro Titanium Internet Security 2013 - Test Winner
    2 Place: Kaspersky Internet Security 2013 - Best in Performance
    3 Place: McAfee Internet Security 2013
    4 Place: Norton Internet Security 2013
    5 Position: G Data Internet Security 2013 - Very good protection
    6 Place: BitDefender Internet Security 2013
    7 Course: F-Secure Internet Security 2013
    8 Place: ZoneAlarm Internet Security Suite

    Visit us- www[dot]antivirus-programme-test[dot]de
    Kiran Choudhary