Inside the belly of the benchmark debate

Inside the belly of the benchmark debate

Summary: David Berlind says Intel's benchmarks don't add up. George Ou disagrees.

TOPICS: Processors

David Berlind says Intel's benchmarks don't add up. George Ou disagrees. The debate on technology benchmarks--whether it's hardware related like Intel vs. AMD or software--is nearly endless.

This Talkback sums up the state of affairs with benchmarks:

When I look at almost any product I can think of, I would not trust the manufactures claims. Instead, I look for independent reviews. There are a number of sites that do real world usage benchmarks (Toms Hardware) and can provide a lot of information on how hardware performs on tasks that you, George, or I might do. That to me means vastly more than some sideways applied counting contest called a benchmark.

Anyhow, yes I agree, manufactures marketing spin in their claims are outlandish, that is why they should be ignored...

The big problem: Technology buyers don't have much beyond manufacturer claims. Benchmarking standards don't exist. Independent third parties are disappearing as tech trade magazines cut back on labs. Reviews are helpful, but benchmarks are still all over the map.

I'm not sure what the answers are to the benchmarking issue, but the problem needs to be fixed across the entire sector. Here's some required reading as we ponder what exactly that fix should be.

Topic: Processors

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • Consumer Reports.

    They do this for a living and seem to know how to make a profit at it. They already review TV's and stereo equipment how much of a stretch would it be to add PCs? And of course that's what real buyers are interested in PCs not just the processor chip.

    How do we talk them into taking this on?
    • They already review PCs ...

      ... but if you think they aren't biased then you haven't been reading their reviews.
  • Real-World Usage...

    is a myth. Even in the personal desktop world, the processor is one piece of a very complex system, whose performance is often dominated by a non-processor component--video card, memory, cache, OS, HD swapping access, algorithm, power consumption. Heck, even NIC-timeout. Enter the server or scientific computing realms, and you're even further from the performance 'truth', as specifics of network and software engineering come into play.

    To top it off, the harder you strive for an 'apples-to-apples' comparison, the further from reality you stray. Do you compare top-of-the-line offerings from both? What if the architecture of the top-of the line differs significantly? What's the metric? Flops? Performance-per-dollar? Performance-per-watt? If the simulated workload differs from the true workload by a magnitude equal to about the performance difference between the offerings what have you really learned?

    You've got two companies, using identical or at most 1-generation apart fab-processes, targeting the same instruction set using, needing to support myriad motherboards.

    It's 98% hype, on both sides. What you have are user requirements, whether its "Will my system run this game at 30fps' or will my webserver crash. If processor A meets those requirements, great. If processor I, meets them, well that's great too. The rest is just a pissing contest. In any event, the final purchasing decision should be based on total cost of ownership, not seeking out a supposed 10-20% advantage from essentially the same technology in usage scenarios you'll never see.


    PS- Where benchmarking IS useful is when you expect to see order-of-magnitude different results, e.g. inter-architecture (RISC vs. CISC, multi-processor vs. virtualization serial), or inter-generational comparisons.
  • Opportunity

    First, let me preface this with the following: I've been a long-time reader of ZD/CNET and respect its authors and contributors, even though I do not always agree with them.

    What I see here is an opportunity for ZD/CNET to do the industry, its readers, and a large group of consumers a service. ZD/CNET offers reviews on thousands of products, and must have some standard way of reviewing things from printers to portable audio devices. Why not do the same for processors and the like in a way that can be comprehended by most consumers?

    Sure, there are sites out there that do this to varying degrees (Anandtech, Tom's Hardware, etc.); however, their reviews tend to be over the heads of those not in the /. crowd. What ZD/CNET could do is tone it down a bit, settle on a few choice benchmarking procedures to test business, home and gaming use. Then, taking advantage of the comparison feature already built into the website, consumers could then stack up each processor, chipset, etc.

    I know, easier said than done. But it would be a great service to the community. I think a key thing here would be to open up the benchmarks to anyone to use. What could distinguish ZD/CNET from the others using the same benchmarks would be quality of writing, presentation, comparison capabilities, etc.

    Just my $0.02.
    • Tone it down a bit?

      For ZDNet, something 'toned down' would be accusing companies of employing child
      rapists while not having a shred of evidence. We all know that the titles of the articles
      on this site are completely different from what is actually in the article, for example
      the numerous 'critical zero-day' bugs that turn out to be nothing. The only thing I
      use this site for is entertainment from ignoramusen like No_ax_to_grind et al, who
      pretend to know everything about what goes on, and really have no more tech
      knowledge than the authors of these inaccurate, biased and sensationalist articles.
  • Usefull benchmarks

    If you don't like the bench marks that other people use, create your own. It doesn't have to be fancy, it just has to give >>YOU<< a metric to compare two or more system. Choose tasks or operations that you use regularly. The tests that you develope only need have meaning to you.

    And yes I have done this. And yes, it did show things that we did not expect to see. And yes, we did base our next IT purchase on these tests. And no, the vendors that tested poorly were very upset.
    • What?

      You can't be serious. What % of the population can actually do this? 2%? Maybe 3%? I'll tell my grandparents to whip up a few benchmarks before they buy their next PC. I love the useless advice that gets posted here. may as well tell a soccer mom to rebuild her own transmission to fix the awful sound it is making. Sheesh....