Android/iPhone web browsing speed test flawed

I'm getting a lot of feedback over a report by Canadian software company, Blaze Software, who have pitted Android against the iPhone in web browsing tests and claim that Android is significantly faster than the iPhone.

Well, you can say that I was curious. So, first off I used the website to run some quick tests, comparing the iPhone iOS 4.2 with the newer 4.3 (for the test I used Google.com as the site to load). I obtained the following results (using the 3 run option):

• iOS 4.2 - 3.88 sec
• iOS 4.3 - 2.67 sec

Hmmm, OK, the browser in iOS 4.3 is a bit faster than the browser in iOS 4.2. That's what I'd have expected from my experience with both versions. But that was just one run. I carried out several runs, and what I found was that I got wildly different results, ranging from under 3 seconds to over 7 seconds no matter which iOS version I used. It seems that the tool is highly susceptible to variation.

I then decided to do some more digging and read Blaze's blog post a little bit closer. I was curious as to how the test was done, in particular how the data was being extracted from the the browsers (especially Safari on iOS, since iOS is a closed box). I came across the answers I was looking for:

The measurement itself was done using the custom apps, which use the platform's embedded browser. This means WebView (based on Chrome) for Android, and UIWebView (based on Safari) for iPhone. Manual verification showed that page load performance of the embedded browsers, when properly configured, is effectively identical to the stand-alone browsers. The load times are calculated using the "Document Complete" callback from the browser, which is a standard way of measuring a web page's load time. As mentioned above, the agents are now a part of a free service available at http://blaze.io/mobile/, and we encourage you to try it out.

Hold on a moment. Blaze is using a custom app. There's a problem with that. As I talked about the other day, Apple is limiting the use of the improved Nitro engine in iOS 4.3 to Safari. Other apps that use the UIWebView controller used by applications to access the web. This could well skew the results significantly.

Given the UIWebView controller issue, combined with the wildly fluctuating results I was getting for one site, I'm going to say that all data puled from this test should be treated as speculative and for entertainment purposes only.

But, even if we assume that the tests are 100% accurate for a moment, is an average load time difference of 1.1 second really going to make that much differenc eis the overall scheme of things when you're dealing with huge variables such as 3G and WiFi connections?

[UPDATE: In the time I was writing this, other have weighed in. Here's what The Loop has to report:

"Their testing is flawed because they didn't actually test the Safari web browser on the iPhone," Apple spokesperson, Natalie Kerris, told The Loop. "Instead they only tested their own proprietary app which uses an embedded web viewer that doesn't take advantage of Safari's web performance optimizations. Despite this fundamental testing flaw, they still only found an average of a second difference in loading web pages."

 It would be a good test ... if it weren't flawed!]

Thoughts?