Microsoft IE9 SunSpider JavaScript results raise questions

Microsoft IE9 SunSpider JavaScript results raise questions

Summary: A Mozilla engineer has uncovered an oddity with Microsoft's Internet Explorer 9 where one test that is part of the SunSpider JavaScript benchmark gives an odd, unexpected result.

SHARE:

A Mozilla engineer has uncovered an oddity with Microsoft's Internet Explorer 9 where one test that is part of the SunSpider JavaScript benchmark gives an odd, unexpected result.

Mozilla engineer Rob Sayre set about benchmarking Firefox 4.0 beta against a selection of other browsers and found that IE9 was about ten times faster at one certain test (the math-cordic test) than the other browsers, with IE9 completing the test in around 1ms while Chrome and Opera took around 10ms.

Curiosity piqued, Sayre did some further investigating:

One last issue that can crop up has to do with over-specialization for a specific test. While I was running the SunSpider tests above, I noticed that IE9 got a score that was at least 10x faster than every other browser on SunSpider's math-cordic test. That would be an impressive result, but it doesn't seem to hold up in the presence of minor variations. I made a few variations on the test: one with an extra "true;" statement (diff), and one with a "return;" statement (diff). You can run those two tests along with the original math-cordic.js file here.

All three tests should return approximately the same timing results, so a result like the one pictured above would indicate a problem of some sort.

This effect shows up nicely in the raw benchmark tests results I carried out last week. Notice how the math-cordic test result for IE9 are consistent.

So what could be behind this. Three possibilities spring to mind:

  • A bug in the JavaScript engine
  • Deliberate optimization for the SunSpider test
  • Accidental optimization for the SunSpider test

Can we put this down to cheating, as suggested by Digitizor (which was later picked up on by Slashdot)? Well, without access to the code it's impossible to be sure, and we don't have access to the code. The effect of this one aberration is quite small and tweaking the values from 1ms to 10ms in the tests I ran only drops the SunSpider score to 403.7ms per run, up from 394.7ms. But this is just one result out of many. It depends if there are other, more subtle, optimizations there.

I'm not ready to call this a cheat yet, but it's certainly fishy. But even is there is some degree of optimization, I'm more likely to believe that it's accidental rather than deliberate. The consistency of the result on IE9 is odd in that across multiple machines I get a consistent score of 1ms, which is not something I'd expect to see. Combine that with the fact that the change made to the benchmark code by Sayre should "functionally" make no difference, the fact that you can see wildly different results is again very odd and not something I'd expect to see.

Sayre has submitted this as a "bug" to Microsoft.

The take away: Benchmarks are odd, fickle things. Put too much faith in the numbers and you lose sight of the wood for the trees.

[UPDATE: It seems some commentators find it hard to see the wood for the trees and wonder why anyone would be suspicious of this test result. I understand the need for nerd-rage diplomacy when dealing with anything Microsoft (pro or anti) but this is about the facts at hand.

Let me sum them up for you:

- Result is consistent across multiple runs (1ms, no variation)

- Result is consistent across multiple platforms - I've run the test on several systems and get a 1ms +/- 0.0% with each and every run.

- The changes made to the code don't functionally change the code (it's hard to put the changes into context, but in this example it's like a car's lap varying by a factor of ten simply by changing the wording on the starting line, that's fundamentally the difference between the JavaScript functions that Sayre tested). It's hard to understand how these changes could add 20ms to the result. Hacker News (via Digitizor) offers some insights as to how this could be a bug.

So it could be a bug, or could be a feature. Either way it's an inconsistency in the code that needs attention just in case it has implications elsewhere. There are many people who make serious business decisions based on benchmark results.]

Ed note: Changed headline of post.

Microsoft had the following to say about the issue:

One of the changes we made to the IE9 JavaScript Engine, codenamed Chakra, to improve performance on real world web sites involves dead code elimination.  Yesterday afternoon, someone posted a question (“What sorts of code does the analysis work on, other than the exact [math-cordic test] function included in SunSpider,”) on the Microsoft Connect feedback site.

Briefly, the IE9 JavaScript engine includes many different changes to improve the performance of real-world Web sites and applications. You can see this in action by visiting www.ietestdrive.com and trying the samples there with IE9 and other browsers. The behavior of the IE9 JavaScript engine is not a “special case optimization” for any benchmark and not a bug.

Some of the optimizations we’ve made to the JavaScript interpreter/compiler in IE9 are of a type known in the compiler world as dead code elimination. Dead code elimination optimizations look for code that has no effect on a running program, and removes the code from the program. This has a benefit of both reducing the size of the compiled program in memory and running the program faster.

The company said it will follow up with more on its IE blog.

[UPDATE: Microsoft has now attributed this anomaly to dead code elimination but this explanation still doesn't account for the fact that the three functions tested by Sayre (and I've run them myself) all include he same amount of dead code ... the addition of true and return statements to the function doesn't in any way change the amount of dead code the JScript engine has to process.

Why the JS engine gives different results for what is functionally the same code with the same amount of dead code is still very interesting and worthy of discussion. The fact that the piece linked to on /. made wild unsupported accusations doesn't change the fact that there's something interesting going on here. Like I said, it's highly unlikely to be cheating but if dead code elimination can achieve such good results for the code used for the math-cordic test, this performance should be translated to the variants using the true and return statements too.]

[UPDATE 2: I've been experimenting with a SunSpider deadcode fork (https://github.com/cheald/SunSpider-deadcode) and the results show that IE 9 does indeed carry out dead code elimination, but that it only seems to kick in under certain circumstances.

There's a lot of dead code in JavaScript out there, you browsers process countless lines of it daily. If Microsoft can make this work more generally, then if would be great stuff all round.

Lot of good discussion here on different code thrown at IE9 being hadled differently ... http://apps.ycombinator.com/item?id=1913368]

Topics: Open Source, Browser, Microsoft

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

73 comments
Log in or register to join the discussion
  • Mozilla "engineer" should look up

    <i>tail recursion elimination</i>. I hope he is not responsible for implementing mozilla JS engine. It would explain some of the stability issues, though.
    honeymonster
    • Talking words without meaning

      He simply put an explicit 'return' in place of an implicit 'return', it has no functional difference, but gee, look at those Benchmarks. It does not change the logic of the returns.

      In his other test he introduced a line of code that should make no difference, yet it makes a huge difference.

      Take note corporate users, nothing has changed, MS products are inherently untrustworthy.
      guihombre
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @guihombre You hit the nail on the head. The difference in the code should make no difference, but it does.
        Adrian Kingsley-Hughes
      • Ignorant bloggers aside

        Function inlining and tail recursion elimination are important optimization techniques. Both rely on the internal structure of a block. JavaScript is a little special because it has a loose syntax (delimiters often not required) and a parser frequently has to "guess" and insert extra delimiters. An artifact of this is that to can inadvertently insert several "null" statements, thus confusing the optimizer to believe that the return statement is <u>not</u> the last statement, and that optimization is not proper.<br><br> In this case the problem may also very well be that the function can be inlined <i>if it has no return statements</i>. A function with a return statement requires more analysis before a compiler can decide whether it can be safely inlined. It is entirely possible that IE9 scans quickly for a return statement before deciding to inline the function. If return statements exists it may very well achieve a better balance in real-world code to just compile and run instead of wasting cycles analyzing the source code. This is a <u>choice</u> compiler implementers have to make, and one they will often make based on samples from the real world. And yes - some choices produce strange artifacts in certain situations, and in some cases these situations arise in synthetic benchmarks.<br><br>You are correct that the statement has no <i>semantic</i> difference and that a compiler <i>should</i> be able to catch this, but JavaScript optimization is so complicated that it is not unfathomable that the extra return statement throws the optimizer off, <u>or</u> that the compiler designers <i>deliberately</i> chose to *not* perform further analysis. <br><br>Is this a undesired effect of the compiler? yes. Should it be able to do better? maybe, depends.<br><br>But you will find such artifacts in <u>all optimizing</u> compilers.<br><br>I don't know how to explain this to ignorant bloggers. They should just go a read up on compilation and optimization techniques.

        Is this a <i>bug</i>? No. Can it be made better? Yes, the special case of a return statement as the last statement of a function should probably be handled as it is not inconceivable that inexperienced developers would write such code.

        But what this boils down to is <b>"incompetent mozilla engineer stumbles on optimization artifact and immediately assumes foul play"</b>. Internet echo-chamber starving for bad news about M$ immediately runs with it.


        <br><br>I'll give AKH this, though, he did not jump directly to conclusions in the article (only here in the talkback). But I also agree with NZ - just putting a question mark after a claim does not absolve you from proper journalistic ethics.
        honeymonster
      • Typical AKH blog using Title to create association between MS and CHEATING

        How pathetic can you be?
        This post could have been made as it would with any other vendor and titled something like IE 9 javascript benchmarks may expose problem.
        But in typical AKH style, pandering to his ABM audience, morals be damned, he goes straight for the "MS CAUGHT CHEATING" which gives anyone who can only gloss over the headlines an indelible impression and serves to enhance his efforts to associate MS with every bad and evil. As with all other ABMers. Surprisingly most are not republicans because I've never seen another group so collectively focused on a group goal, forsaking any bit of thinking for themselves.
        It's easy to create an association such as this by saying it over and over and over again. But you have to be willing to stoop that low.
        xuniL_z
      • honeymonster: The question mark actually DOES make it very clear that it is

        a question. What part of "question mark" do you not understand????????????????????????????????????????<br><br>Now your offhand title of "Ignorant bloggers aside" is NOT excusable. Adrian did a very good job here.
        DonnieBoy
      • Any way you look at it, if MS (and others) start optimizing for the

        benchmarks, then the benchmarks will become meaningless, as they will be no indication of real-world average performance. This happened with video cards in the 90s.
        DonnieBoy
      • @xunil_z: The company that one keeps

        Note that AKH's biggest supporter is DonnieBoy. One can tell a lot about a person by examining the company that they keep. That DonnieBoy is such a friend to AKH tells us a lot.
        NonZealot
      • &quot;You hit the nail on the head. The difference...&quot;

        @Donneiboy<br><br>"You hit the nail on the head. The difference in the code should make no difference, but it does"<br><br>That is what AKH said, and <i>that</i> is which is <b>ignorant</b>. It is ignorant because he clearly assumes he knows enough on the subject to pass judgement, while it is obvious that he doesn't know much about compilers and optimization techniques. This is undergraduate stuff.<br><br>We don't all have to know this, but as a "journalist" blogger he has a responsibility to investigate just a little before participating in which-hunts.
        honeymonster
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @honeymonster
        First of all I would like to thank you for your post. Compiler optimization (tail recursion elimination and other techniques), language semantics and type safety falls in the field of computer science and most of the so called "programmers" have never learnt about it. These topics are taught in the courses like Compiler Construction and Advanced Programming Languages which are offered at Masters or PhD levels. For those who are interested in learning more about it can refer to

        1) Compilers: Principles, Techniques, and Tools by Aho, Ullman and Sethi.
        2) The Formal Semantics of Programming Languages: An Introduction by Glynn Winskel

        There may be many more reference books which deal with these topics but as far as I know these are the standard reference books for many universities, nationally and internationally.

        As a computer science student who has had the privilege of learning about these topics, I appreciate your post and appreciate trying to make people understand how things work.
        1773
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @guihombre Of course - and a 12 year old found a security certificate embedded in Mozilla that no one could account for...
        stevek@...
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @guihombre Wow, talk about blunt!
        jacky.alcine
      • Function inlining and tail recursion elimination

        @honeymonster ... absolutely. Eliminating dead code is incredibly complex, and the initial analysis has to be weighed up against just processing all the damn code. It's highly likely that IE is being conservative with the trade off between compiling the code over messing about with it since over-zealous processing could add more overhead.

        I also agree with you that handling the return statement as the last statement should be handled because it's something that you see a lot.
        Adrian Kingsley-Hughes
      • Explanation Does Not Hold Water

        @honeymonster
        The same result was also achieved by changing the "for" loop in the code to a "while" loop. There is no good explanation for the compiler to react differently to that change except that the code was specifically constructed to recognize the benchmark test and skip it. The only other possibility is that this is an anomaly in the dead code optimization routine that happens to recognize the exact construction of the benchmark by coincidence. That is an awfully big coincidence.

        Microsoft is certainly not the first company to be found to be optimizing their products to score well on benchmarks regardless of real world performance. They will not be the last to do it either. It's not terribly surprising. Of course this doesn't mean that Microsoft is lying about the dead code elimination. It's just that the dead code elimination they are talking about is very specifically targeted at the SunSpider test (wink).

        What this really proves is that benchmarks are not terribly reliable because everyone is shooting for a better benchmark score for marketing purposes. It's always tempting to make optimizations targeting the benchmark rather than general real world results.
        CFWhitman
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @DonnieBoy Sadly, it is a known practice in the area of punditry to phrase an inflammatory/borderline slanderous remark in the form of a question; it gets the idea that there's something bad going on in the collective mind of the public without actually coming right out and saying that it's going on.
        Third of Five
    • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

      @honeymonster Hmm - looking at a few posts, it seems MS have replaced Loverock and NonZealot with someone with a brain! Much better smokescreen and Red Herrings.
      dimonic
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @dimonic

        And Hacker News is smokescreen and Red Herrings for you too I guess

        http://news.ycombinator.com/item?id=1915380

        so IE9 can optimize dead code blocks that's completely different from Sunspider, that clearly means it did NOT cheat in Sunspider.
        hellopeach
      • Not Really

        @hellopeach
        That only proves that their dead code optimization doesn't work all that well (since changing a "for" loop to a "while" loop tripped it up in the SunSpider test).

        You have to realize that Microsoft doesn't consider this cheating at all. As far as they are concerned it's just good marketing. IE9 has dead code optimization. Of course, it's not all that great as far as dead code optimization routines go, but then dead code optimizations are never really all that good (they are attacking the problem of bad coding from the wrong end and trying to come to conclusions that turn out to be fairly heuristic in practice). However, this generally shaky dead code optimization does catch the dead code in the SunSpider test. If there was only one block of dead code that the optimization ever caught it would be one in the benchmark.

        It's the old trap of optimizing your software (or hardware) to score well on benchmarks regardless of real world results. It's not limited to dead code elimination. Whenever the marketing department has a lot of influence in a software company, then if that company has any kind of routine that works really well in only one place, that one place will be on the most widely accepted benchmark. Dead code optimization is just an area that makes this type of thing more obvious.
        CFWhitman
    • Past history

      @honeymonster

      Anyone else remember the C compiler benchmark cheating done a few years back?

      After all, recognizing the benchmark code and replacing it with a table lookup is just good programming technique.
      DNSB
      • RE: Microsoft caught cheating on IE9 SunSpider JavaScript tests?

        @DNSB

        maybe you should get off your blind hate and take a look at this

        http://news.ycombinator.com/item?id=1915380

        so it recognizes completely different codes, and there's no "table lookup" of benchmark codes here, unless you think they have a table entry of "numNumNum" too.

        a human can see that an extra line of "return;" or "true;" as no-op, but many compilers can't, especially JIT compilers. semantically same codes in human eyes can appear completely different to machine compilers. Go learn some stuff before comment in this issue.
        hellopeach