Google's anti-malware team comes out of the shadows

Google's anti-malware team has emerged from the shadows with a new blog, a widely discussed research paper (.pdf) and a few clues about its ambitious drive to put a roadblock between dirty Web sites and end users.

Over the last year, Google has quietly invested in several efforts to flag malicious sites that appear in its search results. Last month, at the HotBots '07 conference in Boston, these efforts came to light when staff engineer Niels Provos (left) released the "Ghost in the Browser" paper with hard numbers on the extent of the malware-on-the-web problem.

In the analysis, Provos and the Google anti-malware team investigated about 12 million suspicious URLs and found that about 1 million of those sites were launching drive-by downloads.

In the paper, Niels dropped a major hint at what's to come from Google:

[We] have started an effort to identify all web pages on the Internet that could potentially be malicious.

The plan has raised eyebrows in some quarters but, as Google's Matt Cutts explained, the company has been working on different ways to warn users about potentially malicious sites. These include an interstitial warning, annotations to listings that a site may be harmful and badware notifications to help Webmasters.

Provos, via e-mail, declined to discuss future plans but there are enough clues to suggest Google is working on some sort of tool to identify hijacked Web servers -- and block drive-by exploits from infecting end-users.

This would put the company up against McAfee's SiteAdvisor, Trend Micro's TrendProtect and Exploit Prevention Lab's LinkScanner, three browser add-ons that slap graphical warning signs (red, yellow or green labels) next to search results.

Provos himself has created SpyBye, an open-source utility that helps Web masters determine if their web pages are hosting browser exploits.

SpyBye operates as a proxy server and gets to see all the web fetches that your browser makes. It applies very simple rules to each URL that is fetched as a result of loading a web page. These rules allows us to classify a URL into three categories: harmless, unknown or dangerous. Although, there is great margin of error, the categories allow a web master to look at the URLs and determine if they should be there or not. If you see that a URL is being fetched that you would not expect, it's a good indication you have been compromised.

Provos told me he created SpyBye on his own time (it's not one of those Google "twenty percent time" projects) to provide a tool for web masters to verify their sites on their own and find out what is wrong with them.

"I wanted to make a tool available that would help web masters discover if their web pages had been compromised to infect users with malware. Many web masters know how to set up and maintain a site, but don't really understand why or how they got compromised. I hope that SpyBye will allow them to get a better understanding of the problem and also allow them to verify if their web pages are still malicious or if the problem has been fixed, Provos said.

Google also has a serious click-fraud problem that is directly linked to botnets of hijacked PCs so it figures that the aggressive anti-malware push will also target bots and Trojans.

It sounds very much like Google could emerge as a player in the anti-virus space. Can a big acquisition be far away?