The BIG browser benchmark (January 2013 edition)

It has only been a couple of months since the last Big Browser Benchmark but there has been enough changes to warrant a re-running the tests. There's also a new benchmark test to put the browsers through their paces.

Let's pit the leading browsers against four of the toughest benchmark tests available to see which one is triumphant. Here are the browsers that will be put through their paces:

• Chrome 24
• Chrome 23 (left in the listing for comparison with newer release)
• Firefox 18
• Firefox 16 (left in the listing for comparison with newer release)
• Opera 12
• Internet Explorer 9 (32-bit)
• Internet Explorer 10 (32-bit)
• Safari 5

Here are the tests that the browsers will face:

• SunSpider JavaScript 0.9.1: A JavaScript benchmark developed by Apple's WebKit team in 2007 with a focus on real-world problem solving;
• V8 Benchmark Suite: A pure JavaScript benchmark used by Google to tune the V8 JavaScript engine;
• Peacekeeper: FutureMark's JavaScript test which stress-tests features such as animation, navigation, forms and other commonly utilized tasks;
• Kraken 1.0: Another JavaScript benchmark developed by Mozilla. This is based on SunSpider but features crucial benchmarking enhancements;
• RoboHornet: A Google-led open-source browser benchmark.
• Octane: Google's new benchmark, based on the V8 test suite.

All testing carried out on a Windows 7 (32-bit) machine, running a P8600 2.4GHz dual-core processor, 4GB RAM, and an NVIDIA GeForce 9400M graphics card.

SunSpider JavaScript Benchmark

SunSpider is a JavaScript benchmark developed by Apple's WebKit team in 2007, and has its focus in real-world JavaScript uses such as text manipulation and encryption. SunSpider is considered by many to be the most comprehensive browser benchmarks available, but as browsers have become faster the tests are now seen as too easy.

The clear winner here continues to be Internet Explorer 10, clearly halving the time it took Firefox 16 and Internet Explorer 9 to complete the test, which themselves are comfortably ahead of the rest of the pack.

While Google seems to have managed to make Chrome a little faster, Firefox 18's performance is slightly worse than that of Firefox 16.

V8 Benchmark

A pure JavaScript benchmark used by Google to fine-tune the V8 JavaScript engine of Google's Chrome browser. The final score is computed from the results of seven demanding tests. This is the tool used by Google to optimize its Chrome browser.

Chrome 24 aces this test, closely followed by Chrome 23, both of which are significantly ahead of Firefox 18 and the rest of the pack.

Peacekeeper Benchmark

A browser benchmark tool from Futuremark, the makers of benchmarking tools such as 3DMark and PCMark. The test covers a lot more than just the JavaScript engine, and gives a good overall view of how fast a particular browser is from the front-facing end.

Google Chrome 23 manages to keep Chrome 24 out of the top spot, with Opera 12 coming in a distance second place -- Opera's best score to date in this series of tests.

Firefox 18 achieves a better score than Firefox 16, but this is a test where Mozilla trails far behind the pack.

Kraken Benchmark

This is Mozilla's JavaScript benchmark tool. A very demanding test that, like SunSpider, focuses on creating realistic browser workloads. 

A close fight between Firefox 18 and Chrome 24, with Firefox 18 coming out in the lead. It seems that both Mozilla and Google have worked hard on their respective browsers to improve their score given the significant improvement.

Internet Explorer 10 holds fifth place, and Microsoft has a lot of work to do if it is to have a chance of beating Mozilla or Google in this test.

RoboHornet Benchmark

This is the new kid on the browser benchmark block. It is Google's vision of a modular, independent, and open-source benchmark comprising of tests created and voted on by developers and designers, with consultation from standards bodies and vendors.

RoboHornet is currently in alpha testing, so it is very much a work in progress and as such the results should be taken with at least a small pinch of salt. That said, it is a test worth keeping an eye on, and is another metric by which to test modern browsers.

Firefox 16 narrowly beats Firefox 18 to the top spot, with Opera 12 in third place. Chrome 23 is in fourth spot, with the newer Chrome 24 also being beaten by its older brother.

Octane Benchmark

This is Google's new benchmark, based on the V8 suite. This is the test suite that Google puts the Chrome browser through.

Google's Chrome browser comfortably beats the competition, with Firefox 18 a distance second. Internet Explorer 10 and Opera 12 are in a distance third and fourth place.

Conclusion

So, with all that, which browser is best?

As with every browser benchmark, it's hard to draw any definitive conclusions from the data given that there's no overall winner. Chrome 24 grabs the top spot in two of the tests, with the other top spots going to Internet Explorer 10, Chrome 23, Firefox 18, and Firefox 16, making statistically the latest version of Google Chrome the overall winner.

That said, the score that Microsoft's Internet Explorer 10 achieves in the SunSpider benchmark is impressive, and it pulls in a respectable score in the other four tests, so it's clear that Microsoft has done a lot of work to make its browser better.

It's also interesting to note that in several of the tests, the older version of the browser beats the new release, which suggests that either Mozilla and Google aren't focusing as heavily on speed as they once were, or that the tests don't accurately reflect they improvements made in the browser.

Having said that, I don't think that JavaScript performance is an issue any more, and certainly when it comes to real world testing it's hard to see a difference between any of the browsers (certain HTML5 sites notwithstanding, given that some are heavily optimized for a particular browser). In fact, unless one of the players managed to boost JavaScript performance by an order of magnitude, shaving a few milliseconds off here and there hardly matters any more.