How effective is endpoint security?

Security software has long been considered a requirement for the Windows environment, whether in the enterprise or at home. Malware cannot only put sensitive information at risk, but can be an expensive nuisance, wasting precious time and bandwidth.

Security vendors claim to protect us against these threats — from a marketing perspective, their products are silver bullets: they'll stop malware in its tracks. When you look at how they perform in the real-world, though, the situation is quite different.

The Race to Zero competition at last year's DEFCON Hacker Conference pitted contestants against a large variety of antivirus products. The winner was the first team that was successfully able to modify supplied malware samples to bypass every antivirus product tested. It took a matter of hours for the winning teams to bypass all antivirus suites for all malware samples.

The current threat landscape

Nowadays much malware is polymorphic, modifying its internal code to automatically carry out its own race to zero, resulting in billions of possible permutations in order to evade detection. Effectively, this means signature-based protection no longer has the same impact it once did, as it simply can't keep up. Other forms of protection used in concert, from heuristics, to whitelisting, to Symantec's new Qourom and Insight technology have become more important; but none are perfect.

It's not only the malware that evolves — the motivations for creating it have changed too. Whereas once upon a time malware was created for notoriety alone, the major impetus today is money. Organised crime funds the efforts of malware authors to find new and inventive ways of getting into your system.

In short: no solution will protect against everything. While installing desktop security software will increase the baseline security of your system, you'll still need to keep your software patched, design your network with security in mind, implement hardware firewalls, and generally follow good security practice.

Common types of malware

There are a few common classes of malware that we'll define for our purposes here, based on how they compromise a system:

Viruses: viruses replicate themselves, usually by infecting files, such as office documents or PDFs. This term is also sometimes used to refer to all malware.

Trojans: trojans are viruses that are sent to the user and require the user to run them. The user may think that the virus is a crucial document that needs reading or a birthday message from a friend or even an illegal crack for software. The file may even open as normal, but in the background a program is run to take control of the system or steal information. Trojans are most commonly associated with email, though they can also be loaded on a website, a USB key, a CD-ROM (using the autostart feature) or even an iPod or camera. Our CANVAS and Metasploit tests used in this article could be classed as trojans.

Worms: these are the big ones that make systems administrators sweat. Worms will run an exploit (or several) against widely used software in order to gain access to a system. Once in, it will continue this process in an attempt to compromise more systems. The Slammer and Blaster worms had a huge impact on the internet, and recently the Conficker worm has been causing enormous issues.

The defence

Enterprise editions of antivirus suites all come with a similar set of features. They generally contain the antivirus component, a firewall/packet filter, web and email scanning capabilities. They also all have a management console you can install on a separate server which will allow for easy administration of all of your antivirus suites, and will use both signatures and heuristics when attempting to detect viruses. Our examination is an attempt to separate the leaders from the followers.

About the Author

Securus Global is one the region's leading Information Security consulting and testing organisations. It has been working with many of Australia's and the world's leading organisations since 2003, consulting and testing business security as well as working with technology vendors to improve the security of their products.

Preparing for testing

For this testing we sourced 2297 pieces of viruses and malware. We attempted to obtain samples as diverse and unbiased as possible. The final assortment of malware included:

Why test Unix files on a Windows machine? A Windows machine may well be the originator of viruses sent to other operating systems, whether through email or another vector. Security stretches beyond just protecting your local machine from the world — it needs to protect the world from your local machine too.

In addition to our assortment of malware we decided to include a number of files from the Metasploit framework and Immunity CANVAS Professional. Both of these tools allow the remote exploitation of systems and come with a huge range of tools and exploits. We use these products in our day-to-day network penetration tests and for custom exploit development.

Not only should these tools be able to show how the antivirus software defends against a targeted attack, but, since the tools use techniques common to viruses and rootkits, they should stress the virus scanners to their limits.

The gritty details

From within Metasploit we created a simple bind shell program, which would bind to a local port when executed and give a shell to anyone who connected remotely. It was then copied via SMB to a VMware virtual machine that had the security suite installed on it, and executed to see if that security suite would pick it up.

We then took it one step further by encoding it in various formats designed to obfuscate and encrypt the program's contents, to bypass any installed security. Using the msfpayload and msfencode programs from Metasploit this is easy to do:

# ./msfpayload windows/shell_bind_tcp LPORT=7878 R | ./msfencode -e
x86/shikata_ga_nai -t exe > bind_nonx_tcp_shikata.exe
[*] x86/shikata_ga_nai succeeded with size 369 (iteration=1)

On the CANVAS side we created a simple callback executable using its BuildCallbackTrojan tool, which is the equivalent of a MOSDEF callback payload that an exploit would use (that is, a program that calls home). In addition to the vanilla callback we created a HTTP downloader which, when executed, would go to a specified URL to download and execute the newly downloaded file.

Using CANVAS we tested to see how the security suite would react to different local attacks such as grabbing passwords, installing malicious drivers and hiding processes to cover up an attacker's tracks.

What does it do to my system?

We also looked at how each of the antivirus tools modifies the kernel and userland. Often antivirus programs use a rootkit technique called hooking. Hooking allows them to intercept and check every call that a userspace program might make to a function in the System Service Dispatcher Table (SSDT) — what Windows looks up to see where in the computer's memory each system function resides.

For example, a Host Intrusion Prevention System (HIPS) program might hook the function ZwCreateFile in the SSDT which is the function that is called every time a file is either opened or created. This would allow the security system to filter access to every file on the system in real time. By examining the hooks that the security software uses we can see what it's likely to intercept, and get a fair idea of its capabilities.

There is a downside to this — the more aggressively you hook functions, the more the performance of your PC is impacted by the security suite, so a balance must be struck, and the right functions hooked.

To examine the hooks we used the program GMER which is, rather ironically, a rootkit detection and removal program. It's named after its Polish author Przemyslaw Gmerek. It boasts a wide feature list detecting everything from hidden processes, threads, modules, services, files, alternate data streams, registry keys, SSDT and other types of hooking.

GMER, in this case showing what McAfee hooks. (Credit: CBS Interactive)

We also looked to see if there were any permission problems using a tool developed by Darren Bilby called Dark Elevator. This simply determined if the installed antivirus software had lowered the security of the system through setting poor file or service permissions, making it easier for exploits to be used.

Dark Elevator, checking out the privilege level of running services. (Credit: CBS Interactive)

Platform/software

The testing was performed using a standard Windows XP Professional Service Pack 3 system, which was fully patched. We created a single common image and cloned it inside VMware Server. Each antivirus suite was installed on the same updated stock build. This gave us full control over the environment to give an accurate comparison.

When necessary we installed the centralised management software on a Windows Server 2003 virtual machine. We also used a laptop with Windows XP Professional installed for additional testing and verification, to ensure that VMware was not affecting the test results.

Testing methodology

We used a single VMware image for each internet security product, and did the following for each product:

• Performed packet capturing with Wireshark: this was used so we could monitor how the antivirus behaves from a network standpoint, especially during its installation/update process.
• Installed and configured the internet security package. The software was installed and configured to be at the appropriate level with all features enabled, including the firewall. Where possible we changed it so that when the AV discovered anything potentially harmful it would delete the file instead of placing it in quarantine. On-access scanning was disabled too so we could safely navigate our folders filled with malware and not have the AV step in and delete everything before we were ready to do the testing.
• Examined how the software works: GMER was run and given an exception to do what it needed (most of the software we tested warned that GMER.exe was doing suspicious behaviour such as loading kernel drivers).
• Firewall testing: tested the average egress filtering rules by using CANVAS callback shells and attempting to reach external ports through telnet and other network executables such as Internet Explorer. Where possible we reviewed the rules manually for anything potentially interesting. External firewall testing consisted of Nmap scans with a wide variety of flags to attempt to bypass the firewall.
• Network load testing: we used various in-house tools to put excess network load on the computers in an attempt to see how well the firewalls dealt with it.
• Default file permissions: after each installation the default permissions were tested on the antivirus folders and files to see if they were correct, as we had experienced permission problems with other penetration tests.
• Malware scan: after disabling the firewall we mounted a share to transfer all the malware to be tested over to the host machine. All malware that was detected was deleted. After testing for all suites, the subset that all antivirus products had detected was taken and packed using a number of publicly accessible packers found on the web (VMProtect, Obsidium, Multi Packer and UPX). Packing is a method of obfuscating the contents of a binary using various methods. This subset was then scanned again by all antivirus software in an attempt to see how effective the public packers were at thwarting detection.
• Attacks using commercial frameworks: using CANVAS we performed various targeted attacks to determine how the suites would defend against a skilled attacker.

Two of these tests provided unremarkable data: Wireshark showed nothing out of the ordinary, and none of the tested software suites suffered from permission misconfiguration — a good thing in both instances.

(Credit: Kaspersky)

Contents

• Introduction
• Methodology
• Kaspersky Total Space Security
• McAfee Total Protection for Secure Business
• Sophos Endpoint Security and Data Protection 9.0.0
• Symantec Endpoint Protection Small Business Suite
• Trend Micro OfficeScan
• ESET Smart Security
• Conclusion

Kaspersky Total Space Security

This product, and in particular the Anti-Virus 6.0 for Windows Workstations component, was quite feature rich in comparison to the other products. Features ranged from the obvious file antivirus to its Host Intrusion Prevention System called "Anti-hacker", which also contains its firewall/packet filter capability.

What does the software change?

Kaspersky was one of the more intrusive antivirus software we examined, in that it hooked 53 functions in the Windows kernel SSDT, which as previously mentioned is the windows system call table. This allows it a much greater control over what it can observe and stop.

In addition to these SSDT hooks, Kaspersky also hooked some functions in the Import Address Tables (IATs) of a number of important kernel drivers such as tcpip.sys. The IAT is used to retrieve the address of functions in other modules, and is often used by rootkits and security software to hook a single function in a driver without hooking that function for every driver in the system.

Scan results

Overall, Kaspersky did well comparatively in the malware scan, missing only 122 files out of the 2297 tested.

Repacked malware

Kaspersky came out on top in numbers in detecting the most repacked malware out of all the products tested, finding 17 out of 27.

Commercial backdoors

Kaspersky did not do well against the commercial backdoors, only detecting the CANVAS HTTP downloader binary (as a generic HTTP downloader). It is likely that this was detected as this binary functions much the same as a lot of malware, which downloads and executes another binary. Malware authors use this technique so the first binary sent to a victim (by email, for example) can be much smaller than the final binary. There are several other reasons a malware author would employ a downloader, such as gaining access to infection statistics and ensuring that new infections occur with an updated second-stage payload.

When attempting to subvert the running system through running the CANVAS callback executable and attempting to load a rootkit and steal password hashes, we were blocked at every stage. The firewall initially stopped our tool from connecting back to the controlling computer, which was expected. We attempted to load a kernel rootkit which was stopped. Recovering from that defeat we tried to steal password hashes by injecting code into the running LSASS.exe process but this too was detected. In addition, the CANVAS binary was flagged so running it in the future generated pop-ups asking if it was OK to run the application as it had attempted malicious behaviour before.

Firewall testing

For some aspects, Kaspersky worked well as a firewall product. It automatically learned what the normal usage was for the computer and built rules that could be manually changed if needed. Once we were happy with the rules we changed it over to the "high security" setting where it would block and allow traffic based off these rules. When a network event that didn't fit into these rules occurred, a pop up would occur asking if we wanted to authorise the program. The manual configuration of the firewall rules was quick and easy to perform.

Where the Kaspersky firewall performed not so admirably was in the network stress tests. When sending large amounts of data to it, the entire machine would freeze for five to 10 minutes at a time, presumably parsing each individual packet. No other software suite tested behaved this way when faced with identical traffic.

(Credit: Securus Global)

(Credit: McAfee)

Contents

• Introduction
• Methodology
• Kaspersky Total Space Security
• McAfee Total Protection for Secure Business
• Sophos Endpoint Security and Data Protection 9.0.0
• Symantec Endpoint Protection Small Business Suite
• Trend Micro OfficeScan
• ESET Smart Security
• Conclusion

McAfee Total Protection for Secure Business

Including McAfee VirusScan Enterprise 8.7i, this McAfee product was interesting in the way that it was broken down into multiple sections, allowing you install the features that your organisation needs. The basic install of the antivirus software did not have any firewall or HIPS support; these features were in the separate "Host Intrusion Prevention" product. In order to configure this product it was required for us to install the McAfee Agent and the McAfee ePolicy Orchestrator (ePO) suite on another system. For example, to configure the firewall rules on the test computer you have to log-in to the ePO web interface on the server system, select the computer and then apply different policies to it.

What does the software change?

Once again McAfee was a bit different from the other examined products in the way that its kernel drivers interacted with the Windows kernel. Instead of actively hooking the SSDT like the others, it uses a technique called inline hooking on the various hooked system calls. This technique has the same end result, although it has some advantages — such as being slightly harder for a malicious attacker to "clean" the kernel of McAfee.

Scan results

McAfee did well, only missing 116 files out of the 2297 scanned. This is the best score of any product in this test.

Repacked malware

We were a little surprised based on the previous result that McAfee did not do well on the repacked malware; it only detected six of the 27 repacked variants. This may indicate that McAfee has a large list of signatures but doesn't do as well on heuristics and variant detection; however, this is just a theory.

Commercial backdoors

McAfee was the only product tested that discovered all of the Metasploit test binaries, including when they had been encoded in various different forms to evade antivirus. This was quite impressive! Sadly, it failed to detect either of the CANVAS files.

When testing the CANVAS attack features, McAfee did not complain when loading up our own driver, which we thought was strange. With a kernel driver we would be able to completely take over the system and disable any antivirus. It did do better when we tried to inject some malicious code into another process, completely stopping it from occurring.

Firewall testing

For the firewall/packet filter testing, we used the "Typical Corporate Firewall" policy setting in the ePO. The default settings were well designed, with nothing out of the ordinary that we could see to quibble over. One thing of note is that McAfee seems to err on the side of caution, making some decisions that suggest that it would rather not break functionality in some networks at the cost of allowing some potentially dangerous services such as SMB and NetBIOS through.

(Credit: Securus Global)

(Credit: Sophos)

Contents

• Introduction
• Methodology
• Kaspersky Total Space Security
• McAfee Total Protection for Secure Business
• Sophos Endpoint Security and Data Protection 9.0.0
• Symantec Endpoint Protection Small Business Suite
• Trend Micro OfficeScan
• ESET Smart Security
• Conclusion

Sophos Endpoint Security and Data Protection 9.0.0

Sophos Endpoint Security and Data Protection had an interesting feature-set in addition to the usual antivirus/spyware related features; it also came bundled with buffer overrun protection, which is becoming increasingly common and important.

The interface to Sophos was designed to look like Windows Explorer, right down to the icons and the way that the application flowed. It even had back and forward buttons, and a couple of times we actually mistook it for an Explorer window. We found the intuitive and plain interface refreshing in comparison to some of the other flashier interfaces.

What does the software change?

Compared to Kaspersky and McAfee, Sophos doesn't intrude a great deal into the system, but still much more than Trend Micro and ESET. Sophos, for example, hooks ZwSetSystemInformation in the SSDT, which is an often used, but non-standard way to load kernel rootkits. This means that Sophos is still able to do some of the major HIPS features such as filtering drivers that are allowed to be loaded.

In addition to the SSDT hooks, it hooks a number of functions in various userland processes such as all the svchost processes, LSASS, and Windows Explorer to implement the previously mentioned "Buffer Overrun Protection".

Scan results

Sophos failed to detect 226 out of 2297 samples, placing it around the middle of the pack in terms of detection.

Repacked malware

In an interesting turn, Sophos managed to detect the more advanced packers, detecting all Obsidian and most of the VMProtect, while missing more of the basic UPX and Multi Packer packed malware. In total it detected 13 out of the 27 repacked files.

Commercial backdoors

In addition to Sophos being the best at detecting the more advanced packers, it was also the only antivirus to detect the CANVAS MOSDEF binary. It also managed to detect the Metasploit bind shell in one of its encoded forms (call4_dword_xor), but not the unencoded binary. Clearly it was detecting the encoder as a potentially malicious item and not the payload.

During testing of the CANVAS rootkit we noticed that while the Sophos HIPS warned that a driver was being loaded, it did nothing to stop it. By default the HIPS is warn only, which we only found out when it failed to block some things we'd assumed it would stop. Once HIPS was configured to block dangerous events, it managed to thwart our attempts at loading future drivers and doing other suspicious actions like injecting into other processes. It even stopped GMER from writing a file that ends with .sys!

Once the software detected our CANVAS binary had been doing some suspicious actions, it flagged it so that every time it was run a pop-up box was created asking if we were sure we wanted to run it, which was a nice feature. Copying the binary and changing the name did not "unflag" the binary.

Firewall testing

The default firewall rules were very restrictive and had to be modified to allow traffic through, which, in our opinion, is how a firewall should be configured when installed. The interface was a little bit clumsy from the menu, but like most of the other antivirus products, it learned rules as programs attempted to open connections. On a positive note this product has a good set of Internet Control Message Protocol (ICMP) rules, such as not accepting ICMP redirects, which can be used in man-in-the-middle connections in some situations.

(Credit: Securus Global)

(Credit: Symantec)

Contents

• Introduction
• Methodology
• Kaspersky Total Space Security
• McAfee Total Protection for Secure Business
• Sophos Endpoint Security and Data Protection 9.0.0
• Symantec Endpoint Protection Small Business Suite
• Trend Micro OfficeScan
• ESET Smart Security
• Conclusion

Symantec Endpoint Protection Small Business Suite

The fourth product we tested was Symantec Endpoint Protection Suite 12.0.122.192. This suite is one of the oldest and most widely used in business, with many of the largest companies using it throughout their networks.

Testing was done on the Small Business Edition of the Endpoint Protection Suite. While it's not the Enterprise version, we were assured by Symantec that the detection and scanning systems are identical to the Enterprise edition.

Initial impressions were quite good. The install process was easy and the user interface quite nice. We scanned our non-infected environment and while it wasn't the fastest scan, it wasn't too bad and came up with the expected lack of results.

When it came to testing our random assortment of viruses, though, it was by far the slowest of all of the scanners. In fact, while the others took from 15 minutes to two hours, Symantec's scan took at least five hours; we ended up leaving it running overnight.

Discussing this with Symantec support led us to discover the cause — when the software discovers malware, it then performs numerous extra checks to ensure that the virus isn't in memory or anywhere else in the system. While this can be a good feature for a home user, it is less applicable in the corporate environment where the computer should immediately be turned off and rebuilt when a serious malware threat is discovered.

What does the software change?

Symantec only hooked the SSDT table and didn't touch any of the kernel code sections. One major difference was that it also attached itself to more devices than the other tested products. Usually, the antivirus would latch onto the "device" that is responsible for IP, TCP and UDP in a bid to monitor traffic but Symantec also monitored the IP Multicast device as well.

Scan results

Symantec, whilst taking the longest to scan, missed 148 of the 2297 of the samples, which places it in third position.

Repacked malware

This software performed well in the repacked malware testing, detecting 16 out of the 27 samples.

Commercial backdoors

Symantec did not do well against the commercial tools, detecting none of them. Even during our CANVAS tests the only alert that was set off was for the connect-back trojan connecting to an odd port. This would not have been detected if it was on a common port, such as 80. We even had to double check that everything was enabled to make sure we hadn't accidentally disabled everything!

Firewall testing

The "Network Threat Protection" offered by Symantec had lax default settings; for example, an attacker wouldn't actually be blocked, the user would simply be presented with a warning. The default firewall rules only blocked IPv6 traffic by default, everything else was open. This can of course be locked down through manual rule sets and other options, and the firewall can be set to either allow all IP-based traffic or narrow it down to application-based traffic, the latter popping up a dialog box asking for authorisation when an application tries to access the network for the first time. This is a remnant of Symantec's buyout of Sygate Personal Firewall, and its network application monitor is here too.

(Credit: Securus Global)

(Credit: Trend Micro)

Contents

• Introduction
• Methodology
• Kaspersky Total Space Security
• McAfee Total Protection for Secure Business
• Sophos Endpoint Security and Data Protection 9.0.0
• Symantec Endpoint Protection Small Business Suite
• Trend Micro OfficeScan
• ESET Smart Security
• Conclusion

Trend Micro OfficeScan

Trend Micro supplied us with its OfficeScan Client-Server Suite 10.0. OfficeScan was different from the other suites in that it didn't have support for an unmanaged client run from the local desktop — everything had to be configured through a central management server over HTTPS.

The install was quite interesting. After installing the server software, the clients could then be installed and configured using an ActiveX control through a web server. The server can also "push" clients to the desktop systems.

The client application was quite spartan, looking similar to a Windows configuration dialog. It only allowed the user to perform manual scans.

What does the software change?

With the exception of ESET, Trend Micro was notably less intrusive than all the other products, only hooking a handful of kernel functions — no userland hooks were discovered.

Scan results

OfficeScan did not detect 447 of the 2297 malware samples. This was by far the worst result of any of the scanners. Disturbingly, some of the viruses that it did detect were marked as "Passed a potential security risk". We gave Trend Micro the benefit of the doubt there and removed those viruses as well to get the final figure, but it didn't bode very well for Trend Micro's detection systems.

A number of the viruses were only detected by turning on Real Time Scanning, and then navigating to the directory. This implied that the real-time scanning was more thorough than the manual scans.

Repacked malware

Continuing in the same fashion as the unpacked scan results, Trend Micro only discovered three out of the 27 repacked malware files. Two UPX (arguably the most well-known packer) binaries and one VMProtected binary were detected.

Commercial backdoors

The scan of the directory containing our commercial backdoors turned up zero results. Trend Micro also did not detect CANVAS at any time during our testing.

Firewall testing

As this is the full enterprise suite, the firewall was extremely customisable, with a strong focus on being able to manually create rules for roles, then applying these roles to the hosts. This contrasts heavily with some of the other applications, which were more focused on automatically detecting normal traffic and generating their own rules.

Sadly, many features you would expect to be found in such a package were not present, such as ARP and DHCP spoofing protection mechanisms, to stop rerouting of network traffic to malicious hosts.

(Credit: Securus Global)

(Credit: ESET)

Contents

• Introduction
• Methodology
• Kaspersky Total Space Security
• McAfee Total Protection for Secure Business
• Sophos Endpoint Security and Data Protection 9.0.0
• Symantec Endpoint Protection Small Business Suite
• Trend Micro OfficeScan
• ESET Smart Security
• Conclusion

ESET Smart Security

ESET's Smart Security product has been making the rounds for a while. It has all of the features you'd expect in the enterprise space including remote administration. The version used here is ESET Smart Security 4.

The client interface was quite interesting, with an advanced mode which was turned off by default. The advanced mode allowed greater configuration and presented some of the higher level features, which was a nice touch.

What does the software change?

ESET had the least amount of kernel changes out of all the products tested, only hooking seven functions, which is a marked difference to a product like the one offered by Kasperksy, which hooked 53. All the functions that ESET hooked were generally related to the ability to monitor processes and threads on the running system.

Scan results

ESET came second last out of our six products, missing 296 of the 2297 sample malware.

Repacked malware

While ESET only managed to discover seven out of a possible 27 repacked malware files, it did manage to detect some of the more difficult to detect packers such as VMProtect.

Commercial backdoors

Like Trend Micro, ESET didn't manage to flag any of our commercial malware. The binaries were undetected and no warning was raised when using the CANVAS node to load drivers or infect processes.

Firewall testing

The ESET firewall and Intrusion Detection System (IDS) features were the richest out of all of the tested products, able to detect known worms and exploits, as well as containing several learning features. The configuration utility had both normal and advanced feature settings, which was a nice touch.

When using the rules provided, all software can make arbitrary egress connections (not a good thing!), such as the one made by the CANVAS trojan.

(Credit: Securus Global)

Conclusion

We were surprised at how much both the functionality and effectiveness varied between these products. Each product uses a different set of functions or resources to perform its activities, and some approaches were radically different than what was expected.

We thought Kaspersky and McAfee were the top performers in our tests. McAfee had the best detection rates in the standard malware scanning, missing only 116 of the viruses. Kaspersky and Sophos, on the other hand, were the only packages that offered the more complete protections against targeted attacks using CANVAS and Metasploit.

There are always ways around this though. In general none of the antivirus packages will protect against a skilled targeted attack, and are mostly ineffective against commercial frameworks such as CANVAS.

It is interesting to note that publicly available packers were extremely effective against the scanners. Kaspersky again performed well against the repacked malware, but even it could only pick up less than two thirds of the malware.

When choosing an antivirus package, though, you need to be aware of your own specific requirements. Make sure you maintain a holistic view of your entire network to ensure that you choose the product that best suits your business.

The following graphs and charts outline the basic differences between the software. It is important to note that due to the random sampling of malware, the total detection figure is not as important as the difference in detection; a small number of these files may not have been actual threats to the platform we performed the testing on.

Effectiveness, normal versus repacked malware

Commercial backdoor performance