HP released a study performed by Sine Nomine Associates (and paid for by HP) claiming that its BladeSystem c-Class servers use 27 per cent less power than IBM's BladeCenter-H models. IBM wasn't fully prepped to respond to the thermal footprint claims, but according to a vnunet.com article Big Blue indicated that the models tested weren't comparable. An IBM spokesperson is quotes as follows: "If they did a like-to-like comparison under their rules and conditions at best it would be a tie."
Last year IBM made similar claims about the energy efficiency of its blades server of HP offerings. As reported last week, Intel and AMD are having some disagreements about the accuracy of benchmarks.
Benchmarks will remain automatically suspect until the vendors come clean and get some agreement on what would be legitimate and fair testing. If they want to have benchmarks, they should agree on the tests and system configurations, and put money into into a pool to pay for independent testing.