Is AMD being hypocritical about benchmark ethics?

Is AMD being hypocritical about benchmark ethics?

Summary: Is AMD being hypocritical when criticizing Intel? We have AMD screaming about Intel’s misleading benchmarks, promises not to do same thing, then appears to turn around gives us outdated scores from a month earlier showing a small AMD advantage when the updated scores actually showed a larger advantage for Intel.

SHARE:
TOPICS: Intel, Processors
118

On Tuesday 2/27/2006, AMD's chief sales and marketing officer Henri Richard swung hard at Intel in front of the media with complaints of unethical usage of outdated benchmarking standards such as SPECint_2000.

Note that SPECint_2000 wasn't officially retired until last week on Feb 24th of 2007 so it was still technically sanctioned to use that benchmark before its retirement and that it isn't Intel conducting these benchmarks but the server makers such as Dell, Fujitsu, and HP.  In either case, it is the most recent Intel score versus AMD that just happens to be on an older benchmarking standard though it can be argued that SPECint_2000 was less relevant than what current applications call for.

Richard aggressively criticized what he characterized as Intel's unethical behavior while promising that AMD will take the high road and not resort to dirty tactics.  In his video interview, David Berlind asked Richard how AMD intends to play this game and Richard responded that "Two wrongs don't make a right and you can expect us to continue to stay true".  Later on in the interview Berlind highlighted one of the benchmarks that AMD was showing at the conference showing how AMD's best dual core Opteron was actually faster than Intel's best dual core Xeon 5160.  See screen capture of Berlind's video interview below.

So here we have AMD’s Richard screaming about Intel’s use of older stats from the previous SPECint standard which shows Intel killing AMD by a large margin, then promises not to do the same thing, then goes right on ahead and offers one month old stats showing AMD’s 3% lead on SPECint_rate2006. Would that be a problem if nothing has changed since that time? No, nothing wrong at all. But I checked with CPU analyst David Kanter of Real World technologies and here’s what he has to say:

In January, AMD lead Intel by about 3% in SPECint_rate 2006; currently, Intel has pulled ahead by around 6%.

SPECint_rate2006 best scores as of Feb 2007

Dual core processor Base Peak
Intel XEON 5160 53.2 55.2
AMD Opteron Model 2220 46.1 51.7
So right now, Intel beats AMD on SPECint_rate for both peak and base.

So we have AMD screaming about Intel’s misleading benchmarks, promises not to do same thing, then appears to turn around gives us outdated scores from a month earlier showing a small AMD advantage when the updated scores actually showed a larger advantage for Intel.  I did a little research and it appears that the updated base scores from Dell were published on February 26th 2007 which is only one day before the presentation so it would be reasonable to let that slide.  But the updated Intel "Peak" scores from Fujitsu Siemens were published by SPEC.org on February 8 2007 which was nearly three weeks before this AMD presentation at the W Hotel on February 27th 2007 so it's a little hard to believe that AMD wouldn't have known about it yet AMD chose to use a one month old slide showing us outdated inferior Intel results.

[UPDATE 3/3/2007 - AMD has notified me via email late Friday that they didn't show the updated Intel scores that were actually 9% higher because their legal team had not approved the updated Intel scores yet.  Since SPEC.org posted scores are about as independent and legitimate as they come and the certification process is extremely rigorous, it’s a bit disingenuous to use your own legal department to block your opponent’s best scores.  The AMD spokesperson told me that they will expedite this legal review process in the future and that “Thanks to you, we will add in a ‘day of’ check, even when presenting a benchmark that was updated only a few weeks earlier.”  I will follow up with AMD on Monday because they want to poke some holes in what Intel presented Wall Street last week.  If it turns out that Intel suppressed better AMD scores in the same way, I'll post that next week.]

But this isn't the only problem with AMD's presentation of benchmarks.  While it might seem reasonable to be comparing the "best" AMD dual-core processor against Intel's best dual-core processor, it completely ignores the fact that Intel customers can choose to save a lot of money with Intel quad-core processors such as the XEON X5355 processor.  Instead of buying two dual-core CPUs, smart shoppers will buy a single quad-core CPU and save tons of money on enterprise server software licensing which is often 5 to 10 times more expensive than the entire server.  While it's possible to save half on the server and slash power consumption with fewer quad core CPUs, it's not unusual to see enterprise software licenses costing in excess of $50,000 per CPU socket!  AMD continues to argue that Intel's quad-core chip somehow isn't "real" because it uses two dual-core dies instead of a single quad-core die.

The problem for AMD is that the benchmarks on Intel's quad-core annihilates any dual-core CPUs so there's nothing "fake" about the performance.  It also happens to be the case that no enterprise software makers charge their software based on die count; they base it on CPU socket count so the fact that Intel has quad-core CPUs and AMD doesn't gives Intel a huge advantage for the first half of 2007.  AMD will probably take back a small edge in the two to four socket server market when their single-die Barcelona by middle of this year but Intel's 45nm Penryn will make sure the honeymoon in Barcelona won't last more than 6 months.  You can see a full AMD versus Intel roadmap here.

It is a well known fact that all companies hardware or software will present their best foot forward and favor the benchmarks that makes them look good.  But leaving out your competitors up-to-date data to make it look like you're ahead in the same benchmark is very disturbing and there is a world of difference between cherry picking benchmarks versus cherry picking data.  The only thing worse than that is hypocrisy.

[poll id=17]

.

Topics: Intel, Processors

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

118 comments
Log in or register to join the discussion
  • AMD does it suck?

    I am a computer consultant and have utilized both procs for many years. Stability and performance have been the ebb and flow with Intel and AMD. What ever the end result, I am most happy with one thing! Intel has serious competition with AMD and visa vi. Nothing better could define our capitalistic market and to me, the consumer, I can purchase a CPU with understanding that I am getting a good bang for the buck.
    muehlbauer
    • Oh Yes it does!!

      I also work for a computer consultantcy company and we make an effort to steer clear of any AMD processors in commercial machines. They simply aren't reliable enough for us to put our trust in them. And the customer has to trust us.
      paddles13
      • Why?

        What failure rate have you observed with AMD processors? What kind of failures have you observed?

        I don't have experience with AMD in large numbers. But, in small numbers, I have NEVER had a failure. Not even one.
        TechExec2
  • You can't blame them for the confusion

    You can't blame them George, they're still battling shellshock. Core 2 shellshock.
    klumper
  • AMD's complaint

    was that Intel's original benchmarks were under suspiciously tightly controlled conditions that were unable to be verified by anyone besides Intel.

    In the example you site, it would appear that AMD used old benchmark figures but they also included the date of the data that they used so it's easy to track where the numbers came from.

    I'm curious as to what differences there are in the SPEC Int numbers from earlier tests on the same CPU. (I thought this SPEC test was aimed at raw CPU throughput so I'm wondering what kind of performance tuning you can do to it.)
    Robert Crocker
    • Where do you make this stuff up?

      "was that Intel's original benchmarks were under suspiciously tightly controlled conditions that were unable to be verified by anyone besides Intel."

      Absolutely not! All the benchmarks in questions were conducted by Server makers and officially submitted to SPEC. AMD was complaining that SPECint_2000 was being used which technically isn't wrong since SPECint_2000 wasn't retired until last week. AMD's more than willing to showcase benchmarks they're better at and it's normal to see some benchmarks favoring one or the other. However, what should NEVER be done is to fake the data by presenting your opponents non-best scores.
      georgeou
      • Didn't see the video

        Didn't realize that it was tied to that particular demonstration and I had remembered the huge hub-bub over Intel's original Core2 benchmarks that were definitely in the "fishy" category at the time.

        I see you've now jumped all the way to accusing AMD of "fak[ing] the data" when in fact they included the date of their information and as you yourself pointed out the other number was produced at a later date.
        Robert Crocker
        • AMD knowingly suppressed better numbers for Intel

          AMD knowingly suppressed better numbers for Intel which were 9% better than the ones AMD presented. If that isn't fraudulent, I don't know what is.
          georgeou
          • George Ou knowingly makes accusations he cannot substantiate

            ...
            TechExec2
          • OMG

            people are bickering over 9% in tests????

            is there anything MORE important in the world going beside meaningless pissant #'s?

            and when are you going to change your signature from georgeou to georgeoe, all MS and Intel all the time???

            i used to read the stuff posted here because there was some useful info, and in the past year it has become a pissing contest

            get over yourselves
            richvball44
  • Marketing is Marketing

    They all lie to promote their product. Some of them even believe in what they sell is purely better. At a time, AMD had a superior product. Now they have trouble admitting that "We've lost." Some of the sales people are still sold on "There is something in the pipe"

    I want to see AMD take more of the market and really level the playing field, but until they release a entirely new architecture or build true Quad Core procs, I don't want to hear about what they have done.
    nucrash
    • *** LATE BREAKING NEWS ***

      [url=http://www.pcworld.com/article/id,129466-page,1/article.html]This[/url] just in.

      And now, back to our regularly scheduled program.

      Back to you George. ;)
      D T Schmitz
      • OK, so now AMD can build a cheaper junk PC

        Oh boy, intergrated chipsets much like Intel, and probably just as powerful.

        I will bet that will help when AMD wants to build a server with a crap video card.

        This does nothing about the benchmarks of existing processors.

        Mr. Dietrich, what you've just said is one of the most insanely idiotic things I have ever heard. At no point in your rambling, incoherent response were you even close to anything that could be considered a rational thought. Everyone in this room is now dumber for having listened to it. I award you no points, and may God have mercy on your soul.
        nucrash
        • OK everybody, just relax

          Hey folks, relax. Don't get all bent out of shape about single digit performance differences between two processors that you don't own. Mr. Dietrich is simply doing what he is paid to do - make his company's product look better than the competitor's.

          "...and may God have mercy on your soul." Just a little over the top, don't you think, nucrash?
          WiredGuy
          • As long as Mike Cox posts...

            I can never be over the top.

            Besides, I was quoting Billy Madison.

            And didn't you hear. Dietrich was cut loose by the recent job cuts at AMD. He is freelance advertising now.
            nucrash
          • Wow.

            Nucrash, you may want to consider an anger management course.

            Add
            addraek
          • That movie was horrible

            Although I was highly impressed with his more recent movie, "Click"
            nucrash
          • Apparently...

            ...either nucrash has had one too many moca lattes at StarBucks or I struck a nerve! ;)

            Mercy!
            D T Schmitz
          • I just didn't care for the Breaking news being over two days old

            Not only that, but also the fact that your late breaking news had nothing to deal with benchmarking what so ever. If you would have told me that Late Breaking news and SPEC released version 2007 or that there were results in for the Quad Cores annihilating the Opteron, or even the fact that the Opteron came out with a Quad core, then I would care. But because you diverted my attention to something that conveys no bearing to the current situation at hand, you ruined my train of thought and ruined my day. I would blame ADD, but today is Friday which means that I should blame you for my problems.

            On a much lighter note,

            Toke Up and Snoogans.
            nucrash
    • good grief

      so should amd close up shop since "they have lost?"
      richvball44