Vista screams past XP in gaming tests!

Vista screams past XP in gaming tests!

Summary: Our technical press, like the mainstream media, sometimes has a hard time letting go of an idea it's been pushing. That's true even when new facts show that the old story wasn't, strictly speaking, accurate. Or when facts on the ground have changed and maybe it's time to alert your readers to the new realities. Today's case in point: A magazine devoted to hardcore gamers and tech enthusiasts does a bunch of tests and proves that Windows Vista with SP1 is every bit as fast as XP, on identical hardware. So what do they do with the story? The answer would make my old J-school professors weep.


Sorry for the exclamation point in the headline, but my inner media critic, who is taking over for the second half of this post, insisted on it. It's there for a reason, which I'll get to shortly, I promise. [Update: My inner media critic is laughing out loud at the people who completely missed this reference and posted TalkBack comments complaining about the headline. "See," he says, "you should have used FOUR exclamation points. Then maybe they would have figured out the headline was deliberately sensational, to make a point about how important headlines and opening pages are to media coverage."]

Our technical press, like the mainstream media, sometimes has a hard time letting go of an idea it's been pushing. That's true even when new facts show that the old story wasn't, strictly speaking, accurate. Or when facts on the ground have changed and maybe it's time to alert your readers to the new realities.

Today's case in point: I just got through reading ExtremeTech's recent lab-based head-to-head comparison of gaming performance in Windows Vista SP1 and XP SP3. The conclusion was quite a surprise.

Here, I'm going to spoil it for you and go straight to the last page. After publishing the results of two synthetic benchmarks (PCMark05 and 3DMark06) and testing frame rates for three popular games (World in Conflict, Supreme Commander, and Crysis), ExtremeTech concluded:

As many a tech analyst predicted, Windows Vista's gaming performance conundrum has largely been solved, and it was mainly due to early graphics drivers.

In fact, I'd been planning to run a few other gaming tests, but the results from these were so uninteresting that further work didn't seem merited. Love it or hate it, Vista is performing far better than it used to.

And the last word?

It took about a year and a half, but the performance gap between Vista and its forerunner has finally evaporated.

Wow! (Insert ironic reference to failed Vista marketing slogan here.) That's news, isn't it? A magazine devoted to hardcore gamers and tech enthusiasts does a bunch of tests and proves that Windows Vista is every bit as fast as XP, on identical hardware. It just needed a year's worth of driver updates and a service pack to get there. (And in the run-up to Vista didn't everyone say you should wait for Service Pack 1? Don't they always say that?)

In fact, ExtremeTech found that Vista actually outperformed XP in three of their six game tests, and scored dead even in the other three. Back when I was a magazine editor, that would have been a cover story with a big, bold cover line:

Vista screams past XP!

A bit sensationalist, perhaps, but guaranteed to get a reader to click through to the end. Or maybe, if you're not willing to give up on the Vista-bashing, you'd prefer this cover line:

Wow! This pig really can fly!

But not at ExtremeTech, where the headline reads like the Estonian lab guys snuck in to the editorial cube farm and wrote it while the editors were out to lunch:

Gaming Performance: Windows Vista SP1 vs. XP SP3

I almost didn't get past the first page of this needlessly chopped-up story (four paragraphs on the opening page, only three paragraphs on page 3, etc.). I guess in the interests of fairness I should insert a page break here. So go, read the second half, in which my inner media critic takes over completely.

Keep reading: Why editing matters -->

Continued from previous page 

My college journalism professors would have flung this story back with a big note reading "Don't bury the lede!" But Author Joel Durham Jr. didn't just bury the lede, he buried the whole story. Instead, the opening page rehashes, with no apparent irony, the industry-standard Vista-is-teh-suxs, XP-is-da-bomb memes:

Even as Microsoft tries to shove Windows Vista down the collective and unwilling throat of computer users worldwide, the company is still perfecting the well-aged and well-loved Windows XP. The latter of the two operating systems just received its third (and evidently last) service pack. ...

Nestled between the bits of Windows XP hagiography on that first page are two more nuggets of Vista-bashing:

Vista, of course, has been plagued by criticism that games run on it don't perform as well as they do in Windows XP...

"Vista, of course, has been plagued..." Nice touch. Everyone knows Vista sucks. Of course.

Though DirectX 10 is Vista-only, many a gamer has sworn not to upgrade...

Sworn while holding a magic sword, no doubt. There is probably an entire level of World of Warcraft devoted to this particular oath.

Anyway, the first page was practically a parody of every I-hate-Vista blog post published in the past 18 months, so much so that I almost didn't bother clicking through the whole thing. And even then it took some effort to continue clicking past Page 2 (a mostly cut-and-paste list of features in XP SP3) and Page 3, which offers a mere three paragraphs (shameful) of "observations" about XP SP3.

It is not until page 5 that the editorial bias begins to crumble, albeit reluctantly, with a pair of graphs and this priceless description:

Vista somehow outperformed XP in PCMark05's overall score, but its important subsystem scores (CPU and Memory) were very close to the older operating system.

That "somehow" is a nice touch. I found that to pronounce it properly one needs to lift one's nose in the air, just so, and then stretch out the syllables in somehow, the way Thurston Howell III might have done it. "Vista somehow outperformed XP." Makes it appear that the actual improvement in performance when running Vista with SP1 on the same hardware as Windows XP is a fluke. Well played, editors!

I encourage you to look at both charts in full, but I found this snippet quite interesting:

 Benchmarks of Windows Vista SP1 versus XP SP3 on gaming

That is a 16.6% improvement for Vista SP1 over XP SP3 in overall performance (the blue bar).

And then you get to the page with the gaming tests and then the final page, which I excerpted at the start of this post.

Unfortunately, ExtremeTech's editing and packaging of this story is really terrible. From experience, I can tell you that the percentage of readers who click through a multi-page post like this one drops with each new page and is depressingly small when you get to page 6 or 7. That means that most people who see the headline and read the first page (or even the first two or three pages) will come away with all their outdated, incorrect stereotypes reinforced. Only those who actually have the patience to make it to the very end will be rewarded with factual information and actual news.

No wonder people believe such crazy stuff.

Topics: Operating Systems, Hardware, Microsoft, Software, Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.


Log in or register to join the discussion
  • I'm not sure Ed

    When reading a performance comparison review like that, the initial few pages of commentary are useless. It is nothing but subjective opinion and at the end of the day, the only real information is contained in the actual benchmarks and sometimes that is represented in the conclusion.

    If you are reading a review like that and you don't actually look at the benchmark results then I suspect you are also the kind of person who runs out to buy Coco Pops after the ad finishes playing on TV.

    Who cares what they say? The data, as long as it isn't tampered with, is all that really matters.
    • I'd beg to differ...

      You're right to say that the most important thing in such a report are the facts and numbers.

      That said, the point here is the fact that the first pages of the report contradict the results presented later. Obviously, the report was written with a prejudiced expectation about how the results would be. And although the numbers later proof those expectations wrong, this text is not changed.

      When i think about possible reasons, there are two coming to my mind:

      - you don't really trust your own numbers, or
      - you're just not willing to give up your prejudices.

      Both reasons are certainly nothing to be proud of for a technical journalist, imho.
      • Technical journalism and vista

        Really they hadn't stated anything meaningful in the first few pages anyway. It just looks like another prejudiced article. Most articles on Vista are like that leaning one way or the other.

        If someone is just going to stop after reading the prejudiced forewords, their mind is already made up with no information. That is really easy to shoot down in a debate when you have actually read the results. Who then will have egg on their face? In this case, (I agree with you about 'nothing to be proud of), the reviewer does look petty when the results are leaning the other way.
    • Dising Coco Pops: Sir you've gone to far!

      The point is performance issues of Vista seem to be improving.
    • People are busy

      These days the average person just scans stuff. They don't go deep. If you chop a big article into seven pages, you just have to assume that the overwhelming majority of people will read only the first page, if they read anything at all.

      That's true times 10 for the A-listers in the blogosphere who are refreshing Techmeme every 15 seconds.
      Ed Bott
      • How bad will they look then...

        when they say "another article bagging Vista"?

        If you are a serious gamer though, the actual performance figures matter more. E.g. if I go Vista SP1 I get DX10.1 but how much performance do I sacrifice? Oh I sacrifice nothing because Vista is better for game X.
  • RE: Vista screams past XP in gaming tests!

    It'll be fun to see which trolls start referencing the extremetech story, only to have their FUD busted by their own link.
  • Very Interesting

    If this holds true, are we possibly going to see the beginning of a turn for Vista? Albeit, that's overwhelmingly hopeful, but good numbers might make many reconsider all the bickering they've done for months. Maybe that's all the pro-Vista-ites can hope for (myself being one).

    Anyhow, you can be assured both sides will have lines to draw on this story, one saying this proves Vista works better, the other claiming that their tests were done improperly and should be discounted. Maybe we can just hope for more follow up studies that show the same.
  • RE: Vista screams past XP in gaming tests!

    I'm awfully suspicious since so many benchmarks have been conducted already on some games that show Vista is no better than XP (except for a few isolated cases). I'd have to see more results than just this article to believe it...
  • Breaking News - Vista as fast as decade old OS!

    Why am I suddenly less than overwhelmed?
    • Now wasn't that to be expected...

      ...first people scream their lungs out at how Vista sucks, and now that it's proven that it doesn't, it's not big news anymore.

      It doesn't get any lamer than that....
      • Save it.

        Read the article and move on. You can comment but don't ever get into fight with them, coz you won't win. Yeah, that's how bad trolling is here.
    • And it took them 1 1/2 years to fine tune it

      to get those numbers (sp1)....<br> How can they get such crappy results from an OS that took basically 7 years to make (two years longer than they expected), the largest public beta test of any Windows version, and 1 1/2 years to fine tune, and its performance is lower or at best equal to what it's meant to replace a almost decade old OS (that people actually like!).
      • No, it didn't...

        It has nothing to do with Vista and everything to do with drivers. It took this long to get quality drivers. Well to be honest ATI's Vista drivers have been great for a long time. Nvidia's have been crap but most sites benchmark with Nvidia cards.

        It's a driver issue, not a vista issue.
  • i never had performance issues with vista...

    i don't use it because of the bugs and dumb interface changes they made. and sp1 did nothing to address either one.
  • Easy does it on ET

    Impressive results for NT6 SP1. Gamers rejoice, long time comimg. But cut the ET boys a little slack Ed. They do good work with their tests, and generally cut things close to the bone while trimming away the fat. They play to the gaming and gearhead crowd in the same way you do to your above-average tech readers: being mindful not to blow smoke lest one is willing to forfeit one's credibility. If they are ever seen to be cheerleading the Redmond gentry and little more, they're as good as sunk with rigmeisters. That's just the way it is.

    I offer the previous two ExtremeTech [i]Vista vs XP Games Reviews[/i], duly dated and freely excerpted for brevity. Now tell me if these guys seem overly biased, or in fact pretty even-handed.

    [b]Will Vista Run Your Games: The Final Word[/b]
    February 5, 2007 | By Jason Cross

    Windows has been the best OS for gamers for years, and Vista takes that to the next level. So let's break it down: Has Vista got game? The basic answer is that yes, Vista is a pretty great OS for gamers. Great, but not perfect. Last May when I looked at beta 2 of Vista, I called it a "B- gaming platform" due to some of the problems I encountered, but that was cutting the unfinished OS some slack for, well, being unfinished.

    The final version of Vista is an A- gaming platform, dragged down to a B- by rough drivers?assuming the driver situation is improved in the coming weeks and months, Vista will be a great gaming OS.

    The long and short of it is that Vista gamers can expect to generally lose a small amount of performance until a few months have gone by and the drivers can be better optimized. I expect this to happen pretty quickly, and for Vista performance to be comparable to Windows XP performance, overall (with some games up to 5% slower, some up to 5% faster).

    [b]Vista Game Performance: Vista vs. XP and ATI vs. Nvidia [/b]
    February 20, 2007 | By Jason Cross

    A couple weeks ago, we took an exhaustive look at Vista gaming by installing over 25 popular titles to see which ones work and which ones don't. Overall, we're pretty impressed with the results. Most games work on Vista just fine, at least as well as they work in Windows XP . On the whole, while Vista is going through the normal teething pains of a new OS, the games compatibility thing seems like a non-issue for most people.

    The catch is, we didn't consider [i]performance[/i] in that article. We know for a fact that some games, even many games, run slower under Vista than they do under Windows XP. It's a whole new graphics driver model, and it's going to take a little more time for Nvidia and ATI to get their drivers fully optimized.

    If you're a gamer, there may be some good reasons to upgrade to Vista, but better performance is not one of them ? and it may never be. Clearly, we're witnessing the same sorts of teething problems we saw when Windows XP was released. We'd like to remind readers that, upon XP's release, many games ran much slower or demonstrated terrible graphics bugs. Eventually, those were ironed out and performance improved until WinXP was just as fast or faster, and more stable, too. We expect the same thing of Vista, and that day can't come soon enough.
    • Valid point...

      You're right, these reports do sound reasonable and well put.

      Perhaps, it was just the one who wrote the report Ed is talking about who's having bit of a hard time overcoming his prejudices...
    • Good post. Better than Bott's bluster. (nt)

      no other text
    • Completely different author and tone

      Those are great. I am assuming (you didn't include links, so I can't check) that those are the introductions. If so, they did a good job of guiding the reader to the story without any inflammatory language. For stories written in February 2007, those feel pretty accurate to me.

      Meanwhile, I stand behind my conclusion that the story I discuss here was terribly edited and packaged, and whoever edited it should go back and talk to whoever edited the two stories you reference.
      Ed Bott
      • The hug-me standard

        Actually they're freely excerpted in the name of brevity, but I tried to make them as balanced and accurate as possible to the overall presentation (mostly by tapping preface and conclusions). For the full articles, here are the links straight from ET:

        [b]Will Vista Run Your Games: The Final Word [/b],1558,2090571,00.asp

        [b]Vista Game Performance: Vista vs. XP and ATI vs. Nvidia [/b],1558,2096940,00.asp

        I should add that I know where you're coming from with the "literary license" you took - for effect - with your subject line. Your titling did what it was intended to: elicit knee-jerk reactions by going a little over the top, much as we sometimes see being done in various Vista reviews. And yes, the editing on the article in question could have been wratcheted a bit tighter. However as I mentioned earlier, hammerheads and enthusiasts are often put off when too much is framed in gold and kisses regarding Microsoft. We have all learned FEW things in the Windows world quite live up to that "hug-me" standard.

        Thus to bring the skeptics and die-hards in, sometimes you have to give a little ground, or concede a point or two as to known or perceived deficits, before seeking to make new strides or cover unexpected ground. Keep in mind, this is the very audience that is the hardest to please (and fool!), because they have plenty of hands-on experience, and thus know all too well the difference between fresh breezes (revelations) and hot air (hyperbole).

        Strangely enough, there's often but a thin line between the two, particularly in matters tech.