Researchers say they've devised a way to bypass protections built in to dozens of the most popular desktop anti-virus products, including those offered by McAfee, Trend Micro, AVG, and BitDefender." - The Register
Sounds familiar? For the past three years or so, such headlines have appeared in the press on a fairly regular basis. This makes one wonder: is there something rotten in the anti-virus world? After all, of all things, we ought to be able to get the security tools right.
Well, the picture is more complicated than it may seem. We intuitively draw parallels between these findings and the vulnerabilities commonly found in other types of software - but the issue here is very different: all consumer-grade anti-virus software simply must be designed to be bypassable, even if only transiently so. To understand this, we need to realize that there is nothing fundamentally different between a botnet client and a legitimate chat application: they both use roughly the same OS facilities in a very similar manner, and while differences are often observed in practice, they are not essential in any way. The intent of the actions performed by these two programs is the only distinguishing factor. Yet, this property can't be algorithmically assessed - not only because the task is notoriously hard to formalize, but also because automated reasoning about the behavior of computer programs is, in many cases, provably impossible.
Because of this, the anti-virus model is nothing like that of "proper" security tools. Instead of eliminating essential mechanisms depended on by the rogue code, AV software tries to stop specific, previously recorded attack patterns: known bad binaries, known suspicious sequences of system calls, and so forth. Some heuristics are employed, but in the end, it always comes down to blacklisting - a canonical bad practice in the field of information security. Surprisingly, in this particular context, it actually works:
Most users are not running up-to-date antivirus software - therefore, there is relatively little incentive for attackers to spend too much time on evading the checks. As a result, most of the malicious code you can run into will trip existing signatures or simple behavioral heuristics. The users of antivirus products remain at a distinct advantage.
Although a percentage of new malware is created specifically to avoid detection, the two possible outcomes are just as favorable to the users of antivirus products:
When malware authors aim for rapid, nondiscriminate propagation, the new variant is quickly captured in the wild, and the tools are updated with new signatures or scan algorithms. In this case, a majority of users are protected before they come into contact with the new payload.
When malware authors want to stay under the radar, and only go after select targets, the majority of users are not interesting enough to end up in the crosshairs - and therefore, the problem is highly self-limiting.
As should be evident, this model works reasonably well to protect casual users. It also tends to fall apart for any entity interesting enough to attract specific attention of determined (but not necessarily skilled!) attackers; for these targets, anti-virus software perhaps reduces operational costs by reducing the likelihood of nuisance infections, but does relatively little to prevent more serious trouble.
Some of the recent reports of antivirus bypass tricks are interesting (this particular research rehashes a problem that all ptrace()-based debuggers and sandboxing tools have to deal with) - but in light of the above, I am inclined to think they do not constitute a security flaw. The attention they nevertheless receive demonstrates that we have only a vague idea of the limitations of AV applications; sadly, with poor understanding, come unrealistic expectations - a major foe of any and all security work.
PS. It is frustrating that contemporary computers are so confusing and vulnerable, that we need such a wonderfully imperfect mechanism to protect the casual user to begin with. That, however, is a wholly different tale.