Vista screams past XP in gaming tests!

Our technical press, like the mainstream media, sometimes has a hard time letting go of an idea it's been pushing. That's true even when new facts show that the old story wasn't, strictly speaking, accurate. Or when facts on the ground have changed and maybe it's time to alert your readers to the new realities. Today's case in point: A magazine devoted to hardcore gamers and tech enthusiasts does a bunch of tests and proves that Windows Vista with SP1 is every bit as fast as XP, on identical hardware. So what do they do with the story? The answer would make my old J-school professors weep.

Sorry for the exclamation point in the headline, but my inner media critic, who is taking over for the second half of this post, insisted on it. It's there for a reason, which I'll get to shortly, I promise. [Update: My inner media critic is laughing out loud at the people who completely missed this reference and posted TalkBack comments complaining about the headline. "See," he says, "you should have used FOUR exclamation points. Then maybe they would have figured out the headline was deliberately sensational, to make a point about how important headlines and opening pages are to media coverage."]

Our technical press, like the mainstream media, sometimes has a hard time letting go of an idea it's been pushing. That's true even when new facts show that the old story wasn't, strictly speaking, accurate. Or when facts on the ground have changed and maybe it's time to alert your readers to the new realities.

Today's case in point: I just got through reading ExtremeTech's recent lab-based head-to-head comparison of gaming performance in Windows Vista SP1 and XP SP3. The conclusion was quite a surprise.

Here, I'm going to spoil it for you and go straight to the last page. After publishing the results of two synthetic benchmarks (PCMark05 and 3DMark06) and testing frame rates for three popular games (World in Conflict, Supreme Commander, and Crysis), ExtremeTech concluded:

As many a tech analyst predicted, Windows Vista's gaming performance conundrum has largely been solved, and it was mainly due to early graphics drivers.

In fact, I'd been planning to run a few other gaming tests, but the results from these were so uninteresting that further work didn't seem merited. Love it or hate it, Vista is performing far better than it used to.

And the last word?

It took about a year and a half, but the performance gap between Vista and its forerunner has finally evaporated.

Wow! (Insert ironic reference to failed Vista marketing slogan here.) That's news, isn't it? A magazine devoted to hardcore gamers and tech enthusiasts does a bunch of tests and proves that Windows Vista is every bit as fast as XP, on identical hardware. It just needed a year's worth of driver updates and a service pack to get there. (And in the run-up to Vista didn't everyone say you should wait for Service Pack 1? Don't they always say that?)

In fact, ExtremeTech found that Vista actually outperformed XP in three of their six game tests, and scored dead even in the other three. Back when I was a magazine editor, that would have been a cover story with a big, bold cover line:

Vista screams past XP!

A bit sensationalist, perhaps, but guaranteed to get a reader to click through to the end. Or maybe, if you're not willing to give up on the Vista-bashing, you'd prefer this cover line:

Wow! This pig really can fly!

But not at ExtremeTech, where the headline reads like the Estonian lab guys snuck in to the editorial cube farm and wrote it while the editors were out to lunch:

Gaming Performance: Windows Vista SP1 vs. XP SP3

I almost didn't get past the first page of this needlessly chopped-up story (four paragraphs on the opening page, only three paragraphs on page 3, etc.). I guess in the interests of fairness I should insert a page break here. So go, read the second half, in which my inner media critic takes over completely.

Keep reading: Why editing matters -->

Continued from previous page 

My college journalism professors would have flung this story back with a big note reading "Don't bury the lede!" But Author Joel Durham Jr. didn't just bury the lede, he buried the whole story. Instead, the opening page rehashes, with no apparent irony, the industry-standard Vista-is-teh-suxs, XP-is-da-bomb memes:

Even as Microsoft tries to shove Windows Vista down the collective and unwilling throat of computer users worldwide, the company is still perfecting the well-aged and well-loved Windows XP. The latter of the two operating systems just received its third (and evidently last) service pack. ...

Nestled between the bits of Windows XP hagiography on that first page are two more nuggets of Vista-bashing:

Vista, of course, has been plagued by criticism that games run on it don't perform as well as they do in Windows XP...

"Vista, of course, has been plagued..." Nice touch. Everyone knows Vista sucks. Of course.

Though DirectX 10 is Vista-only, many a gamer has sworn not to upgrade...

Sworn while holding a magic sword, no doubt. There is probably an entire level of World of Warcraft devoted to this particular oath.

Anyway, the first page was practically a parody of every I-hate-Vista blog post published in the past 18 months, so much so that I almost didn't bother clicking through the whole thing. And even then it took some effort to continue clicking past Page 2 (a mostly cut-and-paste list of features in XP SP3) and Page 3, which offers a mere three paragraphs (shameful) of "observations" about XP SP3.

It is not until page 5 that the editorial bias begins to crumble, albeit reluctantly, with a pair of graphs and this priceless description:

Vista somehow outperformed XP in PCMark05's overall score, but its important subsystem scores (CPU and Memory) were very close to the older operating system.

That "somehow" is a nice touch. I found that to pronounce it properly one needs to lift one's nose in the air, just so, and then stretch out the syllables in somehow, the way Thurston Howell III might have done it. "Vista somehow outperformed XP." Makes it appear that the actual improvement in performance when running Vista with SP1 on the same hardware as Windows XP is a fluke. Well played, editors!

I encourage you to look at both charts in full, but I found this snippet quite interesting:


Benchmarks of Windows Vista SP1 versus XP SP3 on gaming

That is a 16.6% improvement for Vista SP1 over XP SP3 in overall performance (the blue bar).

And then you get to the page with the gaming tests and then the final page, which I excerpted at the start of this post.

Unfortunately, ExtremeTech's editing and packaging of this story is really terrible. From experience, I can tell you that the percentage of readers who click through a multi-page post like this one drops with each new page and is depressingly small when you get to page 6 or 7. That means that most people who see the headline and read the first page (or even the first two or three pages) will come away with all their outdated, incorrect stereotypes reinforced. Only those who actually have the patience to make it to the very end will be rewarded with factual information and actual news.

No wonder people believe such crazy stuff.