PCMark 2005 memory benchmark favors Intel CPUs

PCMark 2005 memory benchmark favors Intel CPUs

Summary: This piece by Ars Technica is, if true, a serious blow for the credibility of Futuremark.

SHARE:

This piece by Ars Technica is, if true, a serious blow for the credibility of Futuremark:

This, gentle reader, is where things get fun. I've heard rumors for years that performance in PCMark 2005 could change depending on what CPUID was handed to the benchmark, but this is the first opportunity I've ever had to test that theory.

...

By changing Nano's CPUID, we can change what value is handed off to FutureMark and expose any irregularities in the benchmark results. If everything is five by five, we shouldn't see any meaningful performance variation at all.

...

My my. Swap CentaurHauls [VIA CPUID] for AuthenticAMD, and Nano's performance magically jumps about 10 percent. Swap for GenuineIntel, and memory performance goes up no less than 47.4 percent. This is not a test error or random occurrence; I benchmarked each CPUID multiple times across multiple reboots on completely clean Windows XP installations.

This is exactly why I dislike benchmark software (I do, I really do - it's up there near the top of my hate list, just underneath Jar Jar Binks). We really need to have a comprehensive open source benchmark tool because current benchmark tools are little more than black boxes fitted with a couple of button that you can click on. There's no way to tell what's going on inside. As the benchmark tool become more and more complex, the scope for a coding blunder increases. I've been quite wary of benchmark scores for some time now, and personally like to have them backed by real world tests too. An open source benchmark would offer greater transparency and credibility.

We need Futuremark to issue a statement on this soon. If this does turn out to be some kind of programming blunder, I hate to think how many purchasing decisions this blunder might have influenced, and how many dollars might have been sent to the wrong company.

Topics: Intel, CXO, Hardware, Processors, IT Employment

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

11 comments
Log in or register to join the discussion
  • Error?

    That just happens to favor the marketshare leader? The same marketshare leader that is up on antitrust violations in several places across the globe?

    Doesn't seem likely. But perhaps I'm too cynical.
    laura.b
  • RE: PCMark 2005 memory benchmark favors Intel CPUs

    Not even the first time that this has happened. Remember BAPCo? They were based out of Intel's headquarter in Germany for years, yet they had the guile to deny any connection. AMD finally joined the BAPCo board, and now we don't hear about BAPCo anymore (Intel has no interest in promoting them anymore). And let's not forget the Ziff-Davis benchmarks from your former sister publication, PC Magazine; those were a suspicious bunch of benchmarks if there ever were such a thing.
    bbbl67
  • RE: PCMark 2005 memory benchmark favors Intel CPUs

    It is a mistake to assume that Windows XP does the same thing with different CPUIDs. This could really be a difference in how XP's scheduler chooses to run tasks...
    jrp@...
    • Windows XP's scheduler?

      Windows doesn't change its scheduling behaviour based on processor. Windows has certain drivers for power management, chipset features, etc., but an x86 processor is an x86 processor, and therefore it schedules everything the same, since the programs don't know the difference.
      bbbl67
    • Didn?

      You know, from back when AMD was beating Intel?

      When XP was released MS was collaborating with AMD (hence the XP in the name). Therefore, it makes little sense for XP to be biased against AMD. Also, given that Intel was always the leader, it makes no sense for XP to be biased against Intel either.
      T1Oracle
      • *please ignore errors I hit enter by mistake*

        The XP stood for Xtreme Performance, but either way AMD did collaborate with MS to develop AMD64. It doesn't make sense that MS would optimize for one over the other except in the 64 bit case where they used the AMD64 code. However, even there the bias should be gone since Intel adopted the AMD64 code.
        T1Oracle
        • collaborate?

          MS did not "collaborate" with AMD on AMD64. In fact, when XP was released, MS calculated its future by accepting Intels plan to extinct x86 in favor for IA64 (itanium). AMD just took the chance to convince everybody, that a fully downways compatible x86 64b architecture would be easier to adopt for the major part of the software industry, namely small software studios. For Microsoft a change of the processor architecture would not have been that dramatical (win NT is designed to allow easy plattform portability from its earliest days), but since even a company as huge as MS is adicted to third party software vendors making software for their platform, they decided to favor x86-64 for it is more wanted by the uncountable hordes of small software studios, which make the windows platform the best choice for most people. (people don't necessarily need the best OS, at first, they need a OS that runs their applications).

          Until late 2004 there was in fact an itanium version of Windows XP, whose distribution was stopped after MS decided in second half of 2004, that they would support x86-64, and would not support two directly competing architectures at the same time. (only since then itanium is an architecture specifically designed for hpc)
          pard
          • Itanium is still ....

            .... supported on the server side. it was dropped from the Windows clients due to lack of adoption by the hardware vendors.
            ShadeTree
        • The XP came from ....

          ... Windows eXPerience not Extreme Performance. XP is not optimised for either nor is the article claiming it is. The author is claiming the benchmarking software is the problem!
          ShadeTree
  • There is one.

    Well... There is the Phoronix Test Suite.
    Anarion
  • Are we really surprised that this be case if true?

    I do not think so. Futuremark has had a track record of these happenings being the case. First the IHVs were called the culprit by providing "optimized" drivers for the tests performed by Futuremark. This however implies that some particular company is funding the benchmark to bias the tests. This isn't the company producing a piece of software that may make the benchmark perform better, a such, but the benchmark to produce bogus results if particular hardware bits were found on the system... Ultimately: Marketing.
    gmureddu@...