Benchmark snafu redux: Gracefully unpublishing benchmarks is impossible to do

Fellow blogger George Ou is hanging on to AMD like a pit bull as a result of what he believes to be deceptive practices in benchmark publishing. In many ways, AMD invited the criticism. Earlier this year, AMD's executive VP and chief sales/marketing officer Henri Richard blasted Intel for misleading the press, Wall St., and ultimately customers with deceptive benchmarks. While Richard's criticisms turned out to be well-founded, they've also drawn ongoing attention to AMD's own benchmarking practices and the resulting probes have not turned out well for AMD.

Whereas I remain unconvinced regarding the extent to which AMD originally engaged in deliberate deception, George cites AMD's failure to expeditiously remove the misleading benchmarks from its Web site (after agreeing to do so) as evidence of AMD's malfeasant intentions (the benchmarks were eventually unpublished today at around 10:45 AM ET).

While the community of benchmark consumers should be thankful for Ou's perseverance, this latest episode of benchmarking drama is just one of many more to come if the industry can't see the forest for the trees. Having spoken with officials from both Intel and AMD, my sense is that both companies are willing to do the right thing when it comes to their benchmark publishing practices. But until everyone is in agreement on what that right thing(s) is/are, the drama will continue. Judging by their actions, Intel and AMD clearly don't agree with each other. But then again, even George and I (theoretically, two watchdogs advocating for the same audience) aren't always in agreement. The idea of unpublishing deceptive benchmarks is a perfect example. George wanted them taken down. I'm not so sure that was the right thing to do.

Unpublishing anything from the Web is impossible. Once something has been published to the Web, it will forever turn up in other places. Not just other Web sites, but in places like Google's search engine cache. If for example, Joe Blogger takes a benchmark chart from AMD's Web site, republishes it in his blog, and refers back to the original page from which he lifted it, by removing the benchmark from that original page, AMD could be creating an even greater problem than if it made some indication that the benchmark was once there, but was since archived for xyz reasons. To some, wiping a benchmark out as though it never existed will make it appear as though the company has something to hide. Might it be better to leave the benchmark online with an honest disclosure? I'm not saying yes. I'm not saying no. I'm saying it's not nearly black and white enough for any one person to be setting the gold standard without some sort of public discourse.

Personally, I don't think removing the objectionable content makes sense. Perhaps the more prudent thing to do is leave it in place and prominently indicate that the chart is out of date. For companies publishing charts on certain home or index pages (as was the case in this episode), this will pose a challenge (do you leave charts on a home page?) which is why, in the spirit of honesty, it makes sense to think hard about how benchmarks are published/referenced on Web pages.

How, for example, do you indicate expiration? If people are going to copy a benchmark for re-use on their own site, how do you enable that (or establish legal requirements) in such a way that shelf-life travels with the graphic anywhere it goes. Do you put benchmarks on home/index pages knowing that you really can't leave them there forever? Perhaps instead, you only point to another page that has the benchmark by itself and that page can reference newer charts when they're available. That page might also be apart of a larger benchmark archive that can always be searched.

Or, as long as a home/index page contains benchmarks, do you always include a history of the charts? For example, at the top of the page, do you put a disclosure like "This page contains or has contained benchmark charts. For a complete history of them, click here." Clicking "here" takes you to the bottom of the page where you'll find a chronological history of the benchmark charts that have appeared on the page with some indicator (eg: an asterisk) indicating which of those are the currently displayed benchmarks. Next to each of the others, you indicate the date that they were retired.

By now, AMD officials are probably saying "Uh... ZDNet... we're getting mixed messages." But that's not the case. Until there's consensus on these issues, we'll be stuck dealing with them tactically. For now, George's demands and AMD's response was probably appropriate given the lack of anything strategic.

Publicizing benchmarks is a very necessary but problematic business and not just when it comes to technologies from AMD and Intel. At any given time, there's a lot of raw data to track, the resulting summaries are scattered across Web sites, PowerPoint presentations, PDF files, etc. and the moral expectations aren't clear. Sure, we expect common sense to help govern the situation. But then, there's a lot of gray area where common sense isn't enough. To absolutely nobody's benefit, this vicious cycle will continue until the industry is ready to work together on the problem.