PCMark 2005 memory benchmark favors Intel CPUs

This piece by Ars Technica is, if true, a serious blow for the credibility of Futuremark:

This, gentle reader, is where things get fun. I've heard rumors for years that performance in PCMark 2005 could change depending on what CPUID was handed to the benchmark, but this is the first opportunity I've ever had to test that theory.

...

By changing Nano's CPUID, we can change what value is handed off to FutureMark and expose any irregularities in the benchmark results. If everything is five by five, we shouldn't see any meaningful performance variation at all.

...

My my. Swap CentaurHauls [VIA CPUID] for AuthenticAMD, and Nano's performance magically jumps about 10 percent. Swap for GenuineIntel, and memory performance goes up no less than 47.4 percent. This is not a test error or random occurrence; I benchmarked each CPUID multiple times across multiple reboots on completely clean Windows XP installations.

This is exactly why I dislike benchmark software (I do, I really do - it's up there near the top of my hate list, just underneath Jar Jar Binks). We really need to have a comprehensive open source benchmark tool because current benchmark tools are little more than black boxes fitted with a couple of button that you can click on. There's no way to tell what's going on inside. As the benchmark tool become more and more complex, the scope for a coding blunder increases. I've been quite wary of benchmark scores for some time now, and personally like to have them backed by real world tests too. An open source benchmark would offer greater transparency and credibility.

We need Futuremark to issue a statement on this soon. If this does turn out to be some kind of programming blunder, I hate to think how many purchasing decisions this blunder might have influenced, and how many dollars might have been sent to the wrong company.