Benchmark snafu redux: Gracefully unpublishing benchmarks is impossible to do

Benchmark snafu redux: Gracefully unpublishing benchmarks is impossible to do

Summary: Fellow blogger George Ou is hanging on to AMD like a pit bull as a result of what he believes to be deceptive practices in benchmark publishing. In many ways, AMD invited the criticism.

SHARE:
TOPICS: Processors
3

Fellow blogger George Ou is hanging on to AMD like a pit bull as a result of what he believes to be deceptive practices in benchmark publishing. In many ways, AMD invited the criticism. Earlier this year, AMD's executive VP and chief sales/marketing officer Henri Richard blasted Intel for misleading the press, Wall St., and ultimately customers with deceptive benchmarks. While Richard's criticisms turned out to be well-founded, they've also drawn ongoing attention to AMD's own benchmarking practices and the resulting probes have not turned out well for AMD.

Whereas I remain unconvinced regarding the extent to which AMD originally engaged in deliberate deception, George cites AMD's failure to expeditiously remove the misleading benchmarks from its Web site (after agreeing to do so) as evidence of AMD's malfeasant intentions (the benchmarks were eventually unpublished today at around 10:45 AM ET).

While the community of benchmark consumers should be thankful for Ou's perseverance, this latest episode of benchmarking drama is just one of many more to come if the industry can't see the forest for the trees. Having spoken with officials from both Intel and AMD, my sense is that both companies are willing to do the right thing when it comes to their benchmark publishing practices. But until everyone is in agreement on what that right thing(s) is/are, the drama will continue. Judging by their actions, Intel and AMD clearly don't agree with each other. But then again, even George and I (theoretically, two watchdogs advocating for the same audience) aren't always in agreement. The idea of unpublishing deceptive benchmarks is a perfect example. George wanted them taken down. I'm not so sure that was the right thing to do.

Unpublishing anything from the Web is impossible. Once something has been published to the Web, it will forever turn up in other places. Not just other Web sites, but in places like Google's search engine cache. If for example, Joe Blogger takes a benchmark chart from AMD's Web site, republishes it in his blog, and refers back to the original page from which he lifted it, by removing the benchmark from that original page, AMD could be creating an even greater problem than if it made some indication that the benchmark was once there, but was since archived for xyz reasons. To some, wiping a benchmark out as though it never existed will make it appear as though the company has something to hide. Might it be better to leave the benchmark online with an honest disclosure? I'm not saying yes. I'm not saying no. I'm saying it's not nearly black and white enough for any one person to be setting the gold standard without some sort of public discourse.

Personally, I don't think removing the objectionable content makes sense. Perhaps the more prudent thing to do is leave it in place and prominently indicate that the chart is out of date. For companies publishing charts on certain home or index pages (as was the case in this episode), this will pose a challenge (do you leave charts on a home page?) which is why, in the spirit of honesty, it makes sense to think hard about how benchmarks are published/referenced on Web pages.

How, for example, do you indicate expiration? If people are going to copy a benchmark for re-use on their own site, how do you enable that (or establish legal requirements) in such a way that shelf-life travels with the graphic anywhere it goes. Do you put benchmarks on home/index pages knowing that you really can't leave them there forever? Perhaps instead, you only point to another page that has the benchmark by itself and that page can reference newer charts when they're available. That page might also be apart of a larger benchmark archive that can always be searched.

Or, as long as a home/index page contains benchmarks, do you always include a history of the charts? For example, at the top of the page, do you put a disclosure like "This page contains or has contained benchmark charts. For a complete history of them, click here." Clicking "here" takes you to the bottom of the page where you'll find a chronological history of the benchmark charts that have appeared on the page with some indicator (eg: an asterisk) indicating which of those are the currently displayed benchmarks. Next to each of the others, you indicate the date that they were retired.

By now, AMD officials are probably saying "Uh... ZDNet... we're getting mixed messages." But that's not the case. Until there's consensus on these issues, we'll be stuck dealing with them tactically. For now, George's demands and AMD's response was probably appropriate given the lack of anything strategic.

Publicizing benchmarks is a very necessary but problematic business and not just when it comes to technologies from AMD and Intel. At any given time, there's a lot of raw data to track, the resulting summaries are scattered across Web sites, PowerPoint presentations, PDF files, etc. and the moral expectations aren't clear. Sure, we expect common sense to help govern the situation. But then, there's a lot of gray area where common sense isn't enough. To absolutely nobody's benefit, this vicious cycle will continue until the industry is ready to work together on the problem.

Topic: Processors

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

3 comments
Log in or register to join the discussion
  • I don't believe what I'm hearing

    "George wanted them taken down. I?m not so sure that was the right thing to do."

    Huh? What do you mean you don't know if it's the right thing to do?

    "Unpublishing anything from the Web is impossible. Once something has been published to the Web, it will forever turn up in other places."

    So what? You take down the ORIGINAL deceptive material that YOU control as a start. So long as the original information is there, it legitimizes that bad information.

    So if someone wrote a really nasty story smearing David Berlind as some kind of felon and it got picked up by Digg and Slashdot as fact, obviously it's going to be a mess undoing the damage with all the sites that have repeated the lie. BUT, would you not want the original site to apologize and IMMEDIATELY take down the bogus information? By your argument David, you seem to be saying that it's hopeless so why bother.
    georgeou
  • There is a BIG difference between a main product webpage and a weblog

    "Personally, I don?t think removing the objectionable content makes sense. Perhaps the more prudent thing to do is leave it in place and prominently indicate that the chart is out of date. For companies publishing charts on certain home or index pages (as was the case in this episode), this will pose a challenge (do you leave charts on a home page?) which is why, in the spirit of honesty, it makes sense to think hard about how benchmarks are published/referenced on Web pages."

    There is a BIG difference between a main product webpage and a weblog. A weblog that has a date at the top of the page that scrolls down to oblivion is one thing, having it prominently on a main product page for an upcoming product is entirely different. The fact that some little birdy sent this page to many news outlets and those news outlets blindly regurgitating these bogus benchmarks makes the timing of this whole thing stink on the part of AMD. AMD can't play the role of the innocent bystander and then refuse to take down the bogus material that is being widely referenced as fact.
    georgeou
  • Like your post, David. One disagreement.

    .
    I like your calm, thoughtful, more circumspect view of the issues with Intel, AMD, and benchmarks. One disagreement:



    [b][i]"...While the community of benchmark consumers should be thankful for Ou?s perseverance..."[/i][/b]

    I disagree. The only benchmark consumers that will be thankful to George Ou are the idiots who don't understand what the word "E-S-T-I-M-A-T-E-D" means. Personally, I think George Ou has been bitten by a rabid Pit Bull Terrier and both need to be put down (1).



    -----------------------------------------

    (1) Debunking George Ou's rabid attack on AMD
    http://talkback.zdnet.com/5208-10533-0.html?forumID=1&threadID=35931&messageID=661568&start=-1
    TechExec2