IE9 takes top benchmark prize, no cheating involved

IE9 takes top benchmark prize, no cheating involved

Summary: I was going to write a short post this morning passing along the news that Microsoft was releasing a new Platform Preview for Internet Explorer 9. And then I got distracted by (I am not making this up) a "slashdotted flamebait troll story." Is it true? Who cares?

SHARE:

I was going to write a short post this morning passing along the news that Microsoft was releasing a new Platform Preview for Internet Explorer 9 (that's number 7 for those keeping score). I had a quick briefing on it yesterday (with an embargo that was to lift when the build was released to the web).

I was probably going to mention that Microsoft was claiming it had made extensive overall performance improvements with this build, including a great deal of work on its Chakra JavaScript engine, and that they had managed to unlock the Top Scorer achievement on the widely followed SunSpider benchmark, ahead of even nightly and experimental builds of the latest and greatest rival browsers.

And I was planning to speculate a little about when Microsoft would deliver a release candidate, which is the next milestone. I was going to guess, given the quick turnaround for this preview release, that an RC will probably be ready before the end of the year.

And then a funny thing happened. Via Twitter, I saw a link to a disturbing headline: ?Microsoft caught cheating on IE9 SunSpider JavaScript tests? Oddly, it led to a post from my colleague Adrian Kingsley-Hughes. But when I started following the original sources, I learned that Adrian had been taken in by a Slashdotted flamebait troll story.

I do not mean those words as an insult or as hyperbole. That is, in fact, the label that the Slashdot editor who posted that story attached to it. Here's the screen capture, complete with mandatory image of Gates-as-Borg.

It's the sort of stuff that Slashdot does well, catapulting an inflammatory story from a tiny website into something its community of ubergeeks can argue about. At Slashdot, the comments are usually more entertaining and informative than the main article.

If you follow the sources back just a few links, you see where this is coming from. Way back in early September, some 10 weeks ago, a Mozilla engineer named Rob Sayre wrote a fairly non-inflammatory blog post giving an update on Firefox JavaScript performance. Given that his blog's tagline is "JS engineering and Kanye West updates," this is not a surprising choice of subject matter. At the end of the post, he noted an odd result for IE9 on one score and speculated that there might be "a problem of some sort" with that result. There is no insinuation of cheating or misbehavior. In fact, it is the last example in a series designed to illustrate some of the small problems that crop up in software engineering.

Nearly 10 weeks later, a tiny blog called Digitizor (based in India, apparently) noticed that post and added its own inflammatory spin. That's the post Slashdot highlighted.

Normally, here's what happens: Slashdot's readers get a big chuckle over the whole thing, the tiny web site goes back to obscurity after its servers take a Slashdot beating, and everyone moves on to the next topic. One positive outcome of the ruckus is that the Mozilla engineer who wrote the original blog post finally got around to filing a bug at Microsoft's Connect site, which is the formal means of raising issues like this so that engineers can look at them in detail. I am authoritatively informed this particular bug is being looked at very carefully.

And there it should have ended. Except that this morning a blogger at ZDNet comes out and repeats the exaggerated allegation, even including the word "cheating" in its headline. ZDNet reaches a big audience. Bigger than Slashdot and orders of magnitude bigger than the little tiny outfit that stirred up this hornet's test. So for ZDNet's millions of readers, an inaccurate story turns into an inflammatory headline.

The original headline on Adrian's post has been slightly updated, with an unconvincing disclaimer tacked onto the end. [By the end of the day the headline had been completely changed, with the word "cheating" removed.] An update at the end of the post itself now contains a response from Microsoft. Unfortunately, anyone who just sees the headline and skims the opener as they scan through their favorite websites will come away with an inaccurate conclusion. [The IE Blog post has now been updated in great detail to address this allegation, complete with an explanation of dead-code elimination as an optimization technique and some actual code samples. If you're a developer, pay attention. If you're not a code jockey, well, your eyes may glaze over.]

The thing that's absurd about this entire incident is that it's obviously untrue. If you try to game a benchmark, you'll get caught as soon as you try to perform the same tasks doing real work. That's especially true when millions of end users have been beating on your beta release for months and developers have been pounding on six previous platform previews stretching back for nearly a full year. Whether you're a user or a developer, you can see the clear improvement in JavaScript performance when you run any of Microsoft's Test Drive examples or visit script-intensive sites.

In fact, I'm about to install that new Platform Preview myself and look at its performance. I was really looking forward to that. It's something I would have started hours ago if I hadn't been distracted by a Slashdotted flamebait troll story. Sorry for the delay. I'll be back with a more detailed look after I've done my homework.

Topics: Software Development, Browser, CXO, Hardware, Microsoft, IT Employment

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

47 comments
Log in or register to join the discussion
  • A few points ...

    My use of the word "cheating" was in reference to the piece picked up on /. and I separate the data that Sayre posted (along with data I'd collected previously) from any finger-pointing.

    Microsoft has now attributed this anomaly to dead code elimination but this explanation still doesn't account for the fact that the three functions tested by Sayre (and I've run them myself) all include he same amount of dead code ... the addition of true and return to the functions doesn't in any way change the amount of dead code the JScript engine has to process. Why the JS engine gives different results for what is functionally the same code with the same amount of dead code is still very interesting and worthy of discussion. The fact that the piece linked to on /. made wild unsupported accusations doesn't change the fact that there's something interesting going on here. Like I said, it's highly unlikely to be cheating but if dead code elimination can achieve such good results for the code used for the math-cordic test, this performance should be translated to the variants using the true and return statements too.
    Adrian Kingsley-Hughes
    • RE: IE9 takes top benchmark prize, no cheating involved

      @Adrian Kingsley-Hughes

      You're confusing an apology with being an apologist.

      You made a mistake Adrian, admit to it and move on.
      tonymcs@...
      • RE: IE9 takes top benchmark prize, no cheating involved

        @tonymcs@... Hey, if I gave the impression of accusing anyone of deliberate cheating, I offer all affected an unreserved and unconditional apology, no problems. But I never supported the idea of deliberate cheating.

        But the issue here isn't cheating, it's about figuring out what's going on here. Experimenting with a SunSpider deadcode fork (https://github.com/cheald/SunSpider-deadcode) would indicate to me that IE 9 does indeed carry out dead code elimination, but that it only seems to kick in under certain circumstances.

        There's a lot of dead code in JavaScript out there, you browsers process countless lines of it daily. If Microsoft can make this work more generally, then if would be great stuff all round.

        Lot of good discussion here on different code thrown at IE9 being hadled differently ... http://apps.ycombinator.com/item?id=1913368

        For me, this issue is interesting, and not some zealot stick I want to wave about.
        Adrian Kingsley-Hughes
      • RE: IE9 takes top benchmark prize, no cheating involved

        @Adrian Kingsley-Hughes

        Yes, dead code elimination is complex, but the problem is that most people don't have a clue how this stuff works, so it is easy to spread FUD when the results don't seem to make sense. Throwing in a single extra line (even if it <i>eventually</i> complies down to a no-op like "true" or "return") can trigger a threshold at which point he compiler no longer even considers the function for further analysis. A JS compiler is much more time constrained compared to an offline compiler (like C/C++/C#/Java etc).

        The problem is that most people don't understand this stuff. For you to go and fuel this FUD is just making the problem worse. I realize people like you and the large amount of ABMers that post in these forums need all the anti-MS FUD you can get. But it is irresponsible to post these kinds of blogs that are designed to create fear, uncertainty and doubt without doing proper research <i>first</i>.
        Qbt
    • RE: IE9 takes top benchmark prize, no cheating involved

      @Adrian Kingsley-Hughes Haven't you, yourself, written articles warning other bloggers of this same type of shotty blogging, Adrian?
      Jared Neale
    • RE: IE9 takes top benchmark prize, no cheating involved

      @Adrian Kingsley-Hughes ... these sensationalist stories are why I stopped reading your blog, Adrian. Stick to hardware. Please.
      GoodThings2Life
      • RE: IE9 takes top benchmark prize, no cheating involved

        @GoodThings2Life If you stopped reading it then how did you make your way into the comments section?
        xSteven777x
      • RE: IE9 takes top benchmark prize, no cheating involved

        @xSteven777x
        Umm, it seems to me that he found his way to the comments section on Ed Bott's Blog, not AKH's.
        Maybe you forgot what story you were reading?
        ke6gwf
      • RE: IE9 takes top benchmark prize, no cheating involved

        @ke6gwf ... yes, exactly right, and kudos for paying attention to details! :)
        GoodThings2Life
    • RE: IE9 takes top benchmark prize, no cheating involved

      @Adrian Kingsley-Hughes

      Mistakes happen just make sure it doesnt happen again. Remember, your's and zdnet's reputation is on the line here, please stay true to being a journalist. Be thorough on your research and be unbiased.

      I dont know how you will feel but it will suck if your name gets associated to shoddy journalism
      DontBeEvil
    • Adrian, you still used the word &quot;cheating&quot; in the title

      @Adrian Kingsley-Hughes
      Using the word "cheating" even if followed by a question mark is highly inflammatory. It is a way to avoid taking responsibility for an allegation you still make. Perhaps that was not your intention, but with your experience you should know this. Until you have substantiated evidence you should refrain from relaying speculative allegations from highly biased sources (./).

      You can do all the testing you want, make all the assumptions you want about how <i>you</i> believe a JS compiler should behave. Testing and reliably reproducing a result does not make you an authority on compiler construction and -optimization.

      A function which returns a value (true) in JS is different from a function which does not (and implicitly returns "undefined"). This could throw a optimizer off. It may be a bug or it may be an artifact. It may even be due to optimizer heuristics and not considered a bug.

      The point is you became an instrument in yet another M$ bashing frenzy. ZDNets and your reputation is on the line here.

      You clearly smelled something fishy, as you stated that you were not prepared to call "cheating" (although it was implied that you were getting there). I hope in the future you will apply your own heuristics and try to get a comment from an authoritative source when that little bell goes off in the back of your head.
      honeymonster
    • RE: IE9 takes top benchmark prize, no cheating involved

      @Adrian Kingsley-Hughes

      Seriously, you should admit you made a mistake and move on. Ed Bott called you out for your mistake. You are not perfect. You were commenting on a subject you have little authority in. You were overreaching. Game over man.

      I commend Ed Bott for holding the line and keeping to a higher standard of journalism.
      surfasb
    • Be careful of standard benchmarks.

      @Adrian Kingsley-Hughes I was working in the Relational Database industry during the benchmark wars in the 1990s. Oracle was able blitz everyone on the TPC benchmarks! It turned out they had written their code to behave differently when it recognized it was running a benchmark, by eliminating most of its internal checks, etc. That way they got great TPC benchmark results, but real word results were no where near that.<br>It would not be too hard to tune your "dead code elimination scanner" to recognize the benchmark code. <br>When the PR begins over browser speeds, it's pretty difficult to dispute published benchmark results, even if they won't reflect real life usage.
      I am Gorby
    • RE: IE9 takes top benchmark prize, no cheating involved

      @Adrian Kingsley-Hughes
      Obviously all Adrian is good at writing is click-bait articles.
      illegaloperation
  • RE: IE9 takes top benchmark prize, no cheating involved

    Of course they didn't cheat, AKH was in too much of a hurry to post an article slamming Microsoft before checking the facts. That is his new routine now. I knew all along that it was just Microsoft optimizing code.
    Loverock Davidson
    • There is nothing wrong with Mr. Kingsley-Hughes'

      story, as it prompted others to research and help eliminate the aligation of the code in question.

      The more of these aligations which are eliminated the better, as [i]Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth[/i].

      In truth, the browser appears to need no help from cheating
      Tim Cook
      • Really?

        @Mister Spock

        "There is nothing wrong with Mr. Kingsley-Hughes'
        story, as it prompted others to research and help eliminate the aligation of the code in question."

        And how many references do you believe we'll see to "M$ cheating!!!!" when they beat the others in the speed tests of the future.

        You are referring loosely to a scientific process where it's ok to put out a hypothesis or a even question. When the recipients recognize it as such it only helps further the discussion.

        But this is not a scientific publication. This is a media for tech-minded readers, not scientists. Trying to hide behind a "question" is the equivalent of CNN asking "Is Sarah Palin really lesbian?" and then reporting on someone somewhere speculating about how she saw her enter a hotel accompanied by 3 other women.
        honeymonster
      • RE: IE9 takes top benchmark prize, no cheating involved

        @honeymonster
        Microsoft have never been beyond cheating in the past so they have a history to correct before anyone takes anything they say seriously (apart from the MS fanbois of course who just suck it up).
        So who can tell if they are or are not cheating - the code isn't visible for anyone to check.
        deaf_e_kate
      • RE: IE9 takes top benchmark prize, no cheating involved

        @deaf_e_kate
        "...the code isn't visible for anyone to check."

        Nor is the code structure of works produced by Adobe, Apple, Oracle (owners of Java) etc; all who've been guilty of cheating and taking shortcuts, yet who fare much more favourably in such questions and reviews.

        My ire doesn't come from the fact that the media and the general tech-based republic holds Microsoft to question as they continue to try and move themselves beyond their less-than-former glories; it is the fact that the same yard-stick is never applied to their contemporaries who played (or continue to play) the same games.
        kaninelupus
  • IE9 Performance

    I've been using IE9 for a while and have downloaded the latest preview. On the whole I like it a great deal, but there is one aspect that puzzles me. Some websites seem to take an absolute age to load for no good reason, that I can see. One I came across this morning is Engadget (www.engadget.com), which freezes for 35 seconds before you can scroll down the screen. Open the same page in Opera, on the same computer, and it's ready in less than 3 seconds. I'd be very curious to know what's going on here.
    TV John