Beyond traditional cyberdefences
Traditional anti-malware tools rely on security vendors having first analysed the malware to extract signatures, which are then regularly updated on customers' systems. But what about new and unknown malware, which could be used by cybercriminals to execute a zero-day attack? Or encrypted/polymorphic malware code that hides its nature and changes each time it runs, avoiding detection by constantly mutating? Counteracting such malware requires a method of detecting potential threats in real time, or near-real-time. This is where automatic malware analysis — and, more generally, next-generation threat protection — comes in.
One style of automatic malware analysis creates a quarantined virtual execution environment (also known as a 'sandbox') that replicates the (usually Windows-based) target for suspected malware and observes its behaviour — extracting details about the payload that can be translated into a signature, and looking for attempts to establish contact with command-and-control servers, for example.
Naturally, cybercriminals are aware of such techniques, and develop ways to detect whether their malicious code is being analysed in a virtual environment, biding its time before infecting the ultimate target. Tricks used by VM-aware malware include looking for human interaction (mouse clicks, responses to dialogue boxes), evading malware analysis schedules, or detecting the characteristic signs of a virtual environment.
FireEye's Enrique Salem believes his company's 'multi-vector execution' (MVX) solution is keeping pace with the bad guys in this respect: "The art for us is that we know how to emulate the user. We do detection on the inbound, and on anything that's trying to communicate. We don't need to know what the attack looks like beforehand: we just need to know the behaviours that are malicious."
Key to the success of this and any other cybersecurity solution is the ability to analyse a wide range of file types and behaviours, and steer an optimal path between identifying false positives (which interfere with legitimate business processes) and false negatives (which leave the organisation open to attack). Security vendors will ideally co-ordinate the (anonymised, metadata-based) threat intelligence gathered from their customers and partners in a cloud-based repository, creating a positive feedback loop into the overall threat protection system.
Other solutions target different levels of the IT stack — from monitoring network packets and flows for suspicious behaviour, up to application controls. To be truly effective, a next-generation threat protection system needs to be able to collate multi-faceted intelligence — about spear-phishing emails, suspicious files, contact with external command-and-control servers and anomalous flows of data (often encrypted) out of the network, for example — to build up a complete picture of an advanced persistent attack and defeat it.
These next-generation threat protection solutions (leading vendors of which are covered below) form an additional line of defence on top of traditional firewalls, intrusion prevention systems, secure email/web gateways and endpoint protection solutions. It's important to note that both approaches are required to deliver an acceptable level of security: think of traditional anti-malware tools as uniformed police chasing and apprehending known villains, and the next-generation solutions as plain-clothes detectives seeking shadowy wrongdoers who have yet to acquire a criminal record.
Next-generation threat protection: approaches and vendors
Next-generation threat protection solutions typically come as custom-built rackmount appliances that sit inside an organisation's network and inspect inbound and outbound email, web and file-share traffic, and also files at rest, for suspicious behaviour or characteristics in near-real/real time. Cloud-based services are also available to do a similar job for smaller businesses that don't want to invest in the high-performance hardware required, although they'll need to be aware of potential issues with file-type coverage, scalability and data protection.
Research firm Gartner has recently made a useful classification of approaches to next-generation threat defence, based on a 3-by-2 matrix of where to look (at network traffic, malware payloads or endpoints) and the timescale involved (near-real/real time or post-compromise):
This leads to five styles of advanced threat defence, based on the intersections of the rows and columns (the payload/post-compromise cell clearly being irrelevant):
Network Traffic Analysis
Analysing network protocol and/or content traffic in real time allows security professionals to establish a baseline for 'normal' activity so that anomalous patterns can be detected. Leading vendors in this area, according to Gartner, include Arbor Networks, Damballa, Fidelis Cybersecurity Solutions, Lancope and Sourcefire (now part of Cisco).
Incident response teams, in particular, need access to network forensics tools that perform full-packet capture and metadata extraction, and provide sophisticated analytics and reporting capability, along with high-capacity storage. Leading vendors in this area, according to Gartner, include Solera (a Blue Coat company) and RSA NetWitness.
This is where the aforementioned sandboxing solution, which can reside in an on-premise appliance or (with caveats) in the cloud, comes in. Malware behaviour is observed and characterised in near-real-time, capturing threats that are missed by signature-based tools. Leading vendors in this area, according to Gartner, include: AhnLab, Check Point (ThreatCloud Emulation Service), FireEye, Lastline, McAfee, Palo Alto Networks (WildFire), ThreatGRID and Trend Micro (Deep Discovery).
Endpoint Behaviour Analysis
Although it can be an operational headache, endpoint behaviour analysis (in the form of application virtualisation and containment, system configuration monitoring, memory monitoring, process monitoring and application whitelisting) can offer protection to mobile devices that are off the enterprise network. There are potential issues in the shape of OS support and device resource usage though. Leading vendors in this area, according to Gartner, include: Blue Ridge Networks, Bromium, Invincea, Sandboxie and Trustware (application containment); Cyvera, ManTech/HBGary (Digital DNA) and RSA Ecat (memory monitoring); and Triumfant (system configuration & process monitoring).
On-device data-collection agents can help incident response teams characterise malware attacks, but this type of solution does not block attacks as they occur, and places another heavy operational burden on the IT team. Leading vendors in this area, according to Gartner, include Bit9, Carbon Black, Guidance Software (EnCase Analytics), Mandiant and ManTech/HBGary (Responder Pro).
As Gartner points out, an optimal next-generation threat protection strategy will usually involve at least two of these 'styles' — for example network traffic analysis plus network forensics, or payload analysis and network forensics.