Tech

Google's Project Zero fuzzed top browsers for bugs: Safari users won't like the results

Google's Project Zero releases the open-source tool it used to find new bugs in major browsers.

Written by Liam Tung, Contributing Writer Sept. 22, 2017 at 3:29 a.m. PT

Project Zero said most browsers did relatively well in the tests but Apple Safari is a clear outlier.
Image: Google Project Zero

Google's Project Zero has open-sourced an automated security testing tool that helped find 31 security bugs across all major browser this year, but most of them were in Apple's Safari browser.

If you read Apple's security update documents each month, a name that has regularly appeared this year under bugs patched in Safari's WebKit engine is Project Zero's Ivan Fratric.

The tool he used to find them is a fuzzer called Domato, which has exposed 17 now-patched bugs in Safari. It also found bugs in Chrome, Edge, and Internet Explorer, but far fewer.

Domato is designed to uncover bugs in the DOM or Document Object Model engines of Chrome, Safari, Edge, and Internet Explorer. The DOM engines are part of each browser's rendering engine, such as Chrome's Blink engine, which Google created in 2013 after forking WebKit.

As Fratric points out, DOM engines are a major source of browser bugs, and are occasionally targeted by attackers, such as the zero-day exploit in Firefox found to be used against users of the Firefox-based Tor Browser last November.

That exploit was accidentally leaked by Exodus Intel and reportedly was used to identify visitors to a child pornography site.

Along with revealing the browser bugs his tool has helped find, Fratric has now open-sourced Domato in the hope that others can improve it.

Each browser was fuzzed -- which involves throwing random code at it in an attempt to cause crashes -- around 100 million times. Since a single machine would be too slow at this, the browsers were tested from Google's cloud.

According to Fratric, fuzzing at this scale on Google's Compute Engine would cost about $1,000, which a determined attacker could easily afford.

The good news for all browsers, except Safari, is that Domato's results suggest no browser's DOM engine is better or worse. Fratric found two in Chrome, four in Firefox, four in Internet Explorer and six in Edge.

However, according to Fratric, Safari's performance was concerning so, before today, he offered Apple access to Domato, which was eventually taken up when a Project Zero member was hired by Apple.

"Apple Safari is a clear outlier in the experiment with a significantly higher number of bugs found. This is especially worrying given attackers' interest in the platform as evidenced by the exploit prices and recent targeted attacks," he writes.

The other notable finding is that the gap between Safari and Chrome's DOM-related bugs widened after Google forked WebKit.

"It is also interesting to compare Safari's results to Chrome's, as until a couple of years ago, they were using the same DOM engine (WebKit). It appears that after the Blink/Webkit split either the number of bugs in Blink got significantly reduced or a significant number of bugs got introduced in the new WebKit code (or both)," he notes.

"To attempt to address this discrepancy, I reached out to Apple Security proposing to share the tools and methodology. When one of the Project Zero members decided to transfer to Apple, he contacted me and asked if the offer was still valid. So Apple received a copy of the fuzzer and will hopefully use it to improve WebKit."

He also gave props to the Microsoft team that created MemGC or Memory Garbage Collector, an exploit mitigation feature it built for Edge and Internet Explorer 11.

"It is also interesting to observe the effect of MemGC, a use-after-free mitigation in Internet Explorer and Microsoft Edge. When this mitigation is disabled using the registry flag OverrideMemoryProtectionSetting, a lot more bugs appear," he writes.

"However, Microsoft considers these bugs strongly mitigated by MemGC and I agree with that assessment. Given that IE used to be plagued with use-after-free issues, MemGC is an example of a useful mitigation that results in a clear positive real-world impact. Kudos to Microsoft's team behind it."