My use of the word "cheating" was in reference to the piece picked up on /. and I separate the data that Sayre posted (along with data I'd collected previously) from any finger-pointing.
Microsoft has now attributed this anomaly to dead code elimination but this explanation still doesn't account for the fact that the three functions tested by Sayre (and I've run them myself) all include he same amount of dead code ... the addition of true and return to the functions doesn't in any way change the amount of dead code the JScript engine has to process. Why the JS engine gives different results for what is functionally the same code with the same amount of dead code is still very interesting and worthy of discussion. The fact that the piece linked to on /. made wild unsupported accusations doesn't change the fact that there's something interesting going on here. Like I said, it's highly unlikely to be cheating but if dead code elimination can achieve such good results for the code used for the math-cordic test, this performance should be translated to the variants using the true and return statements too.
I was going to write a short post this morning passing along the news that Microsoft was releasing a new Platform Preview for Internet Explorer 9 (that’s number 7 for those keeping score). I had a quick briefing on it yesterday (with an embargo that was to lift when the build was released to the web).
I was probably going to mention that Microsoft was claiming it had made extensive overall performance improvements with this build, including a great deal of work on its Chakra JavaScript engine, and that they had managed to unlock the Top Scorer achievement on the widely followed SunSpider benchmark, ahead of even nightly and experimental builds of the latest and greatest rival browsers.
And I was planning to speculate a little about when Microsoft would deliver a release candidate, which is the next milestone. I was going to guess, given the quick turnaround for this preview release, that an RC will probably be ready before the end of the year.
And then a funny thing happened. Via Twitter, I saw a link to a disturbing headline: Microsoft caught cheating on IE9 SunSpider JavaScript tests? Oddly, it led to a post from my colleague Adrian Kingsley-Hughes. But when I started following the original sources, I learned that Adrian had been taken in by a Slashdotted flamebait troll story.
I do not mean those words as an insult or as hyperbole. That is, in fact, the label that the Slashdot editor who posted that story attached to it. Here’s the screen capture, complete with mandatory image of Gates-as-Borg.
It’s the sort of stuff that Slashdot does well, catapulting an inflammatory story from a tiny website into something its community of ubergeeks can argue about. At Slashdot, the comments are usually more entertaining and informative than the main article.
If you follow the sources back just a few links, you see where this is coming from. Way back in early September, some 10 weeks ago, a Mozilla engineer named Rob Sayre wrote a fairly non-inflammatory blog post giving an update on Firefox JavaScript performance. Given that his blog’s tagline is “JS engineering and Kanye West updates,” this is not a surprising choice of subject matter. At the end of the post, he noted an odd result for IE9 on one score and speculated that there might be “a problem of some sort” with that result. There is no insinuation of cheating or misbehavior. In fact, it is the last example in a series designed to illustrate some of the small problems that crop up in software engineering.
Nearly 10 weeks later, a tiny blog called Digitizor (based in India, apparently) noticed that post and added its own inflammatory spin. That’s the post Slashdot highlighted.
Normally, here’s what happens: Slashdot’s readers get a big chuckle over the whole thing, the tiny web site goes back to obscurity after its servers take a Slashdot beating, and everyone moves on to the next topic. One positive outcome of the ruckus is that the Mozilla engineer who wrote the original blog post finally got around to filing a bug at Microsoft’s Connect site, which is the formal means of raising issues like this so that engineers can look at them in detail. I am authoritatively informed this particular bug is being looked at very carefully.
And there it should have ended. Except that this morning a blogger at ZDNet comes out and repeats the exaggerated allegation, even including the word “cheating” in its headline. ZDNet reaches a big audience. Bigger than Slashdot and orders of magnitude bigger than the little tiny outfit that stirred up this hornet’s test. So for ZDNet’s millions of readers, an inaccurate story turns into an inflammatory headline.
The original headline on Adrian’s post has been slightly updated, with an unconvincing disclaimer tacked onto the end. [By the end of the day the headline had been completely changed, with the word "cheating" removed.] An update at the end of the post itself now contains a response from Microsoft. Unfortunately, anyone who just sees the headline and skims the opener as they scan through their favorite websites will come away with an inaccurate conclusion. [The IE Blog post has now been updated in great detail to address this allegation, complete with an explanation of dead-code elimination as an optimization technique and some actual code samples. If you're a developer, pay attention. If you're not a code jockey, well, your eyes may glaze over.]
The thing that’s absurd about this entire incident is that it’s obviously untrue. If you try to game a benchmark, you’ll get caught as soon as you try to perform the same tasks doing real work. That’s especially true when millions of end users have been beating on your beta release for months and developers have been pounding on six previous platform previews stretching back for nearly a full year. Whether you’re a user or a developer, you can see the clear improvement in JavaScript performance when you run any of Microsoft’s Test Drive examples or visit script-intensive sites.
In fact, I’m about to install that new Platform Preview myself and look at its performance. I was really looking forward to that. It’s something I would have started hours ago if I hadn’t been distracted by a Slashdotted flamebait troll story. Sorry for the delay. I’ll be back with a more detailed look after I’ve done my homework.
