Sun accused of cheating Java benchmarks

Pendragon Software Corporation, developer of the widely-recognised CaffeineMark, yesterday said a Sun Microsystems Java compiler identifies part of CaffeineMark to produce a misleading score.

An October 20 Sun Microsystems press release claimed Sun's Web-enhanced Solaris operating system runs Java faster than any other environment.

It states: "The new Web-enhanced Solaris 2.6, with integrated Java technology, provides the best foundation for companies to run Java applications at significantly higher speeds than Windows NT."

The release also says: "The new Solaris record, 50 per cent faster than Windows NT, is a result of benchmarks conducted by Sun using CaffeineMark 3.0 ... and Solaris 2.6 software."

Pendragon Software analysed the results and discovered that CaffeineMark's Logic Test gave the Solaris JIT compiler a score 50 times higher than any rival thus far. "There were three possible explanations for this result: either the compiler was performing some impressive optimisations, there was a bug in the compiler, or there was a bug in the benchmark," Pendragon's press released response claims.

As a control, Pendragon made what it claims are minor syntactical changes that had virtually no effect on other platforms but made Sun's score 300 times lower. Pendragon concluded the compiler was pattern matching against the benchmark, "and performing optimisations that do not work in the general case".

Further inspection showed a block of 600 bytecodes that exactly match part of a file used in CaffeineMark.