Futuremark reverses--Nvidia didn't cheat
"Futuremark now states that Nvidia's driver design is an application-specific optimization and not a cheat," said Futuremark in a statement.
Nvidia and Futuremark issued a carefully-worded, fence-mending joint statement on Tuesday. Explaining why it decided to drop its allegation that Nvidia was fudging results, Futuremark said its normal practice was to frown on optimized code, but said it may change this practice.
"3DMark03 is designed as an un-optimized DirectX test, and it provides performance comparisons accordingly. It does not contain manufacturer-specific optimized code paths," the company said. "Because all modifications that change the workload in 3DMark03 are forbidden, we were obliged to update the product to eliminate the effect of optimizations identified in different drivers so that 3DMark03 continued to produce comparable results.
"However, recent developments in the graphics industry and game development suggest that a different approach for game performance benchmarking might be needed, where manufacturer-specific code path optimization is directly in the code source. Futuremark will consider whether this approach is needed in its future benchmarks," the company said.
Last week Futuremark, one of the leading benchmark software providers, alleged that Nvidia tweaked software needed to run its new GeForce FX 5900 processor to distort performance in Futuremark's 3DMark03 testing application.
The company said drivers--software files that govern how a component interacts with the rest of the PC--for the new Nvidia chip were altered to detect activity characteristic of a benchmark and adjust performance accordingly.
Benchmarks can be important to the prestige, if not the market share, of graphics chipmakers. Nvidia is in the midst of a hard-fought battle with rival ATI to claim the performance lead in PC graphics processors. After years of Nvidia dominating both in market share and performance, ATI took the speed lead last year with new versions of its Radeon chips.
After settling manufacturing issues that resulted in ongoing delays of product launches, Nvidia hoped to regain the performance lead this year with the 5900.
In the mean time, new ZDNet lab tests show that the scores of Nvidia's latest hardware dropped by between 14 percent and 22 percent using Futuremark's updated testing software, although at higher image-quality settings the Nvidia chip still matches the performance of ATI Technologies' latest Radeon. The update of Futuremark's 3DMark03 eliminates the effects of Nvidia's optimizations. (See the full review: Nvidia GeForce FX 5900 Ultra: updated benchmarks.)
When ZDNet testers originally ran 3DMark03 tests, using Build 320, Nvidia's 5900 Ultra ran circles around the ATI 9800 Pro 256MB. Under the updated Build 330, the 5900 Ultra's 3DMark03 scores dropped by between 14 percent and 22 percent, depending on the resolution and the feature settings, testers said.
At both 1,024 by 768 and 1,600 by 1,200, with no advanced features enabled, the Radeon 9800 Pro 256MB is now noticeably faster. However, once anti-aliasing and anisotropic filtering are enabled, the performance of the two cards is almost identical.
Results remained mixed for other tests, as they had been under the original Build 320 results, and testers said that neither card could be said to have taken a decisive lead. However, that is an improvement for Nvidia, whose previous GeForce FX 5800 Ultra did not measure up to the competition.
ZDNet UK's Matthew Broersma and CNET News.com's David Becker contributed to this report.