/>
X

Intel trots out misleading benchmark charts

In a manipulative attempt to demonstrate leadership over AMD for the press and analysts, Intel twists benchmark data to make it look as though it's latest Xeons are outperforming the best AMD has to offer. Yet, upon closer inspection, most of the data being shown is irrelevant.
zd-defaultauthor-matt-conner.jpg
By Matt Conner, Contributor on
59009.png
1 of 7 Matt Conner/ZDNET
In a manipulative attempt to demonstrate leadership over AMD for the press and analysts, Intel twists benchmark data to make it look as though it's latest Xeons are outperforming the best AMD has to offer. Yet, upon closer inspection, most of the data being shown is irrelevant.
Using line charts to present benchmark data is tricky to begin with. Usually, you have performance points going up the Y axis and along the X axis, you might have variable loads. But you'd never do what Intel has done here.
In this graph designed to show leadership in energy efficiency, not only do the benchmarks used change from one data point to the next, but so too do the chips being compared (as evidenced by the extensive footnoting). Whereas the Intel chip stays constant (a dual core Xeon 5160), the AMD chip to which it's compared changes from one benchmark to th next. In some cases, it's the newest chip. In others, an older generation Opeteron is used. Still others, Intel is comparing its dual core to an AMD single-core
To compound matters, for two of the benchmarks that show Intel taking a commanding lead, both of the benchmarks are six year old benchmarks for which newer versions exit. In fact, of the last ten benchmarks to the right side, all but one compare the Xeon 5160 to a chip it shouldn't be compared to, or use a retired benchmark.
Where the little yellow question mark appears, we don't even know what AMD chip was used there. It doesn't say. Maybe it was an old 32-bit Athlon?
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.
59010.png
2 of 7 Matt Conner/ZDNET
This slide, taken from a presentation give one week after the previous slide appeared, is a slight improvement in that Intel is calling out single vs. dual core comparisons. But even so, the rest of the problems remain.
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.
59011.png
3 of 7 Matt Conner/ZDNET
Although we don't have all the annotations on this slide like we did on the first, we could have simply taken them from the first slide and overlayed them onto this one. Pretty much every benchmarking felony committed on the first slide in this presentation was committed on this slide as well, thus completely calling into question Intel's characterization of its performance as "Breakaway." Based on this data, it's nothing of the sort. Would Intel consider the Opteron's advancements over its old 386 technology a sign of a breakaway?
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.
59012.png
4 of 7 Matt Conner/ZDNET
In one version of this slide, Intel cites two SAP benchmarks in declaring "leadership benchmarks" of it's latest greatest Xeons over AMD. But the footnotes reveal that the SAP performance differences are over AMD's older family of Opterons (the 885). In fact, they're the 2.6Ghz 885's which weren't even the fastest versions of those chips. AMD had a 2.8Ghz version as well.
As if this slide doesn't have enough problems, it too cites the 6 year old SPEC benchmarks for which newer versions have bene issued by their authors.
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.
59013.png
5 of 7 Matt Conner/ZDNET
In this version of the previous slide (the version used to show the press and analysts on February 28th), the fine print that shows how the SAP 4P benchmark was not run against the latest greatest Opteron (the 2200SE) is missing. With that footnote missing, it would be easy for the press to assume it was against the newest generation of Opteron processors. But in reality, as can be seen fro the previous slide, it was not.
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.I
59014.png
6 of 7 Matt Conner/ZDNET
We were shocked when we saw this benchmark chart. There were so many footnotes to denote how the chip being compared to was not an Opteron 2XXX (as the headings say) that we wondered why not do it the other way around? Instead of saying which chips were not an Opteron 2XXX, footnote which ones were. There would have been fewer footnotes!
Check out the top right hand quadrant of this chart. The heading says "vs. Opt 2XXX." Yet, not one single system under that was an Opteron 2XXX. We were imagining the press and analysts looking at this slide, not paying attention to any of the footnotes to realize that there wasn't a single shred of truth t the heading.
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.
59015.png
7 of 7 Matt Conner/ZDNET
In this slide, Intel even had the audacity to use big bold text that says "Continued Leadership vs. Latest Opteron." Yet, 21 of 32 results shown don't involve the latest Opteron or even a member of th the latest Opteron family. And again, just like with the last slide, the top right hand quadrant of data has "vs. Opt 2XXX" in the heading. But none of the chips listed below it were an Opteron 2XXX. Everyone of them had to be footnoted.
For David Berlind's write up on Intel's incredibly misleading usage of benchmarks, see his post in ZDNet's TestBed blog.

Related Galleries

Hyundai Ioniq 5 and Kia EV6: Electric vehicle extravaganza
img-8825

Related Galleries

Hyundai Ioniq 5 and Kia EV6: Electric vehicle extravaganza

26 Photos
A weekend with Google's Chrome OS Flex
img-9792-2

Related Galleries

A weekend with Google's Chrome OS Flex

22 Photos
Cybersecurity flaws, customer experiences, smartphone losses, and more: ZDNet's research roundup
shutterstock-1024665187.jpg

Related Galleries

Cybersecurity flaws, customer experiences, smartphone losses, and more: ZDNet's research roundup

8 Photos
Inside a fake $20 '16TB external M.2 SSD'
Full of promises!

Related Galleries

Inside a fake $20 '16TB external M.2 SSD'

8 Photos
Hybrid working, touchscreen MacBook hopes, cybersecurity concerns, and more: ZDNet's tech research roundup
Asian woman working at a desk in front of a computer and calculator

Related Galleries

Hybrid working, touchscreen MacBook hopes, cybersecurity concerns, and more: ZDNet's tech research roundup

8 Photos
Developer trends, zero-day risks, 5G speeds, and more: Tech research roundup
Person seated at a booth in a cafe looks at their phone and laptop.

Related Galleries

Developer trends, zero-day risks, 5G speeds, and more: Tech research roundup

10 Photos
Drive Electric Day: A dizzying array of EVs in sunny Florida
ca3b4019-26c5-4ce0-a844-5aac39e2c34b.jpg

Related Galleries

Drive Electric Day: A dizzying array of EVs in sunny Florida

16 Photos