Antivirus tests need better methodology

While most antivirus tests do not have the right assessment methods to determine their effectiveness, vendors cannot afford to ignore them as they are key to branding.
Written by Ellyne Phneah, Contributor

Antivirus assessment studies may lack the right methodology to determine how effective antivirus products are, but vendors still need to take them seriously as they are key to branding.

According to Simon Piff, associate vice president of enterprise infrastructure and research at IDC Asia-Pacific, an effective antivirus is able to catch all types of malware, not just a sample of malware.

However, the test's selection process will always affect the results since it is no longer "all the malware in the universe", but a "representative sample", which is unable to fully reflect the universe of malware accurately, he explained.

Most tests too, do not have an accurate methodology in measuring the effectiveness of antiviruses, Peter Stelzhammer, vice chairman of testing firm AV-Comparatives, added. The most accurate test of antivirus efficiency is passing it through a real-life scenario with a statistically valid number of test cases, he noted.

Work on malware detection, don't be intrusive to users
Aside from scoring well in antivirus tests, vendors should also continue innovating and improving their products to detect malware better, because malware evolves at a fast rate and millions of new viruses are introduced everyday, Piff noted.
The most important thing an antivirus should do is protect a user without having too much impact to the system's performance or "annoying" users with too many pop-ups, messages or questions.
In response to a call for comments, Twiiter user @gmanka said he wanted "something that actually protects your PC and isn't intrusive while it's at that". Another consumer @asian_angel said she wanted something easy to set up and use, reliable and did not eat up a lot of system resources.
Singaporean consumer Liang Wen Min however, said she did not mind her software being intrusive as long as it did its job.
She explained her antivirus had been "useless" despite not compromising her PC's performance and left her systems infected with ransomware.

Antivirus tests stir controversy
Their comments come after effectiveness tests conducted on antivirus products in December stirred controversy among security vendors.

Security vendor Imperva released a study in December, where it used an online tool VirusTotal to pit antivirus products against 82 randomly collected malware samples, examining how successful they were in detecting them. It was found from the study anti-malware software was not fast or responsive enough to combat targeted threats, and security software is better at detecting malware that spreads rapidly in massive quantities of identical samples.

German testing firm AV-Test too, put 25 antivirus products for their November to December 2012 study, and found only 92 percent of the zero-day attacks were blocked during the test. The products were able to clean 91 percent of the infected systems, but only 60 percent could be put back in a condition similar to the pre-infection state, the firm said.

Antivirus vendors have retaliated against these studies, attacking the test methodologies. Microsoft for one, told ZDNet last week the use of malware in tests did not reflect real-life conditions, and their customers did not encounter the malware samples used. Microsoft's Security Essentials had only achieved 71 percent in November and 78 percent in December for AV-test's study.

Responding to the Imperva study, Kaspersky Labs also told PC Mag, the VirusTotal service did not use the full versions of antivirus products, but relies merely on a standalone scanner. "This approach means the majority of protection technologies available in modern antivirus software are ignored. This also affects proactive technologies designed to detect new, unknown threats," Kaspersky Labs said.

Explaining the test methodology in an earlier report, Andreas Marx, AV-test's CEO said the focus of the test had been on samples from major malware families. Even though getting hit by these samples are low, getting hit by the families of these samples are high, and an antivirus that is able them has demonstrated effectiveness

Important for branding
That said, antivirus tests are not completely irrelevant as it can prove a good branding opportunity, Stelzhammer pointed out.

When antivirus vendors score well across a variety of tests, they prove their quality and performance, which is way more effective than claiming to have the best product on the market, he explained.

BitDefender for example, was named product of the year in AV-Comparatives' test in December last year. "We feel that it improves consumer perception of our antivirus and moving forward, we will strive to ace more antivirus tests for better branding," the antivirus vendor said.

Editorial standards