XP SP2 vs. Vista RTM vs. Vista SP1: Gaming benchmark

XP SP2 vs. Vista RTM vs. Vista SP1: Gaming benchmark

Summary: Here are the benchmark results that many of you have been waiting for - a look at how well ten popular games work on XP SP2, Vista RTM and Vista SP1.

SHARE:

Here are the benchmark results that many of you have been waiting for - a look at how well ten popular games work on XP SP2, Vista RTM and Vista SP1.

It's been nearly a year since I took a look at the state of gaming under the Vista OS.  The last time I compared XP SP2 to Vista, I was disappointed with Microsoft's new OS as a gaming platform because I could get far more bang for my bucks out of XP than I could out of the new OS.  Partly I put this down to immature drivers, but on the whole I was convinced that at the core of the problem was Vista. 

So, is Vista worth bothering with as a gaming platform?

The tests

The tests will be carried out on the AMD Spider platform that I have set up in the lab (Phenom 9700, Radeon 3850 graphics card, 2GB of RAM …).  I’ve used this system as the platform for a number of benchmarks I’ve run over the past week (for a full spec, see this post).

Here's the deal.  I set up three images for the system - one based on XP SP2, one on Vista RTM and one on Vista SP1 (all 32-bit flavor).  On each of these images all the updates provided by Microsoft were installed.  The installation of drivers in addition to those supplied by Microsoft was kept to a minimum (XP SP2 required more drivers than Vista).  On each of the images the latest ATi Catalyst drivers (version 8.2) were installed.  Then, in order to be able to measure and record frame rates, the latest version of the FRAPS utility was installed.

Finally, the games we were going to were then installed and each one updated to the latest version.  Here's a list of the games used:

  • BioShock
  • Call of Duty 4
  • Company of Heroes
  • Crysis
  • Doom 3
  • F.E.A.R.
  • The Elder Scrolls IV: Oblivion
  • Supreme Commander
  • Unreal Tournament 3
  • World in Conflict

For these tests we chose not to rely on any of the canned benchmark tools supplied with some of these games.  Instead we relied on real-world play - that is, playing the same bit of the game on each of the platforms (a technique that HardOCP use when benchmarking).  While this method is subject to variations, if you've played games for long enough and are familiar with the map, you can pull this off quite well.

Note: Why not used canned benchmarks?  Well, by not doing so we are ruling out the possibility that the graphics card drivers that we use (or even the OS) has in any way been optimized to produce better results for the canned benchmarks that you might see in the game. 

To come up with workable graphics settings we first experimented with each of the games in turn.  When we found graphics settings that worked well (all the tests were run at 1280 x 1024 resolution in full-screen mode with the games set to either medium or high quality graphics) these settings were carefully recorded replicated them across the three platforms.

So to come up with the average frame rate (measured in frames per second) we play the same bit in each game with FRAPS recording, repeat this four times.  If there was any wild variation in the data we went back and took another stab at it (we only had to do this once, with Supreme Commander).  Once we were happy with the data we averaged it out to get a final frame rate score. 

We were very pleased with the end data because we ended up with runs where the average frame rate only varied by one or two frames per second.  This gave us confidence in the final results we achieved.

The results -->

The results

These tests took longer to carry out than I'd planned - in fact, the results that follow are the fruits of over a week of work.  Still, to get a hands-on snapshot of the current state of gaming on both XP SP2, Vista RTM and Vista SP1, it was worth the time and effort.

Here are the results. 

The table below shows the average frame rates for each game on each platform.  We also show the percentage difference compared to XP SP2 (so, for example, a value of -10% indicates a drop in frame rate of 10% compared to XP SP2, which 10% would indicate a 10% increase in frame rate compared to XP SP2).

XP SP2 vs. Vista RTM vs. Vista SP1: Gaming benchmark

Here are graphs showing average frame rates for the ten games across all three platforms:

BioShock

Call of Duty 4

Company of Heroes

Crysis

Doom 3

F.E.A.R.

The Elder Scrolls IV: Oblivion

Supreme Commander

Unreal Tournament 3

World in Conflict

Conclusions -->

Conclusions

Because we're juggling with a lot of data here, I'm going to try to present the conclusions as clearly and concisely as possible.

  • On the whole, applying SP1 to Vista makes little difference to most games.
  • Seven out of ten of the games tested produced frame rate averages which were lower under Vista than XP SP2.
  • However, a closer look at the numbers shows that when Vista is slower than XP SP2, the actual frame rate differences are only in the single digits.
  • The only exception to the above is Doom 3, which shows a drop of 19 frames per second on Vista RTM and 15 frames per second on Vista SP1.  This is the largest variation we saw during testing.
  • The three games which performed better on Vista than on XP SP2 were Call of Duty 4, F.E.A.R. and The Elder Scrolls IV: Oblivion.
  • Vista gave The Elder Scrolls IV: Oblivion the best performance boost compared to XP SP2, working out at an average of 9 frames per second faster on Vista RTM and 13 frames per second faster on Vista SP1.

 So, what's the bottom line?  Well, XP SP2 is still the better OS when it comes to gaming.  However, Vista is getting better.  Thanks to several performance updates from Microsoft, better drivers from hardware vendors and updates from the game makers themselves (a good example of this improvement is Oblivion, which was awful on Vista when it went RTM but has improved to the point where it's actually better on Vista than on XP).  What would be interesting to try (although I don't have the time to do this) would be to re-run the benchmarks with a bare, unpatched Vista RTM and compare this to Vista SP1 - my guess is that you'd see significant improvement.  Still, even comparing Vista SP1 to XP SP2, it's clear that Vista has a long way to go before it can beat its older sibling.

Still, the fact that some gamers work better on Vista means that those after the maximum frame rates possible might want to set up dual boot systems.

As always, your mileage WILL vary!

Thoughts?

Note: Here are links to previous Vista benchmark posts: 1 - Some systems showing incredible SP1 performance boost | 2 - Vista SP1 vs. XP SP2 - Benchmarked | 3 - Vista SP1 vs. XP SP2 - Part Deux | 4 - Vista 32-bit vs. Vista 64-bit - Benchmarked | 5 - Vista: 32-bit vs. 64-bit & RTM vs. SP1 | 6 - Vista RTM vs. Vista SP1 - Office 2007 benchmarking.

<< Home >>

Topics: Operating Systems, Microsoft, Software, Windows

Kick off your day with ZDNet's daily email newsletter. It's the freshest tech news and opinion, served hot. Get it.

Talkback

70 comments
Log in or register to join the discussion
  • I wonder

    Of those games, how many are optimized for DirectX 10?
    No_Ax_to_Grind
    • Here are the ones I know

      Bioshock supports DirectX 10, but ZDNet is testing it in DirectX 9.

      I think CoD 4 is DirectX 9.

      Company of Heroes is DirectX 9, but has a patch for DirectX 10.

      Crysis is DirectX 10.

      Doom 3 is DirectX 9.

      F.E.A.R. is DirectX 9.

      The Elder Scrolls IV: Oblivion is DirectX 9.

      Supreme Commander is DirectX 9.

      Unreal Tournament 3 I'm not sure about.

      World in Conflict I'm not sure about.
      CobraA1
      • Doom3 and FEAR are OpenGL games

        NT
        pa2004
        • True

          But they use DirectX for the setting up the window etc. hence their requirement for DX9 - if you try installing either on a <DX9 system it will want to update it in the install program.
          Ben_E
    • All were benched in DirectX 9 mode ...

      ... but we did try both Crysis and BioShock in Dx10 on Vista - and Crysis suffered an FPS crisis while BioShock delivered exactly the same numbers.
      Adrian Kingsley-Hughes
      • Thanks for the info

        Was just curious.
        No_Ax_to_Grind
        • DX9, DX10, DX10,000

          Vista is still CRAP.
          iwillnotbestopped
          • W.O.W. I think Vista retailers are alway a problrm; no return no refunds @

            30 day return policy...This hurts me to hav a young person off of XP SP2. Vista can really be fun with some years of over spending on computers at a living wage. I think we lost one the the larest 29" bicycle or a new dirtbike 'Star' at the track; those are great outlooks for next Christmas.
            rtirman37
      • What about OpenGL? [nt]

        nt
        olePigeon
  • sigh . . .

    "Well, XP SP2 is still the better OS when it comes to gaming. "

    . . . except there's no DirectX 10.

    When I look at the scores, it's really not a big difference. A game that runs at about 30 fps is still running at about 30 fps, and a game that runs at about 40-50 fps still runs at about 40-50 fps. A game that runs at 100+ fps (Doom 3 is really showing its age) still runs at 100+ fps.

    Quite frankly, they're all staying within the ballpark as far as frame rates go. This really proves that ZDNet is making a big fuss about nothing.
    CobraA1
    • "making a big fuss about nothing"

      That also applies to those who have fanatically supported XP and avoided Vista. Whether gaming or business use, Vista has performed very well for me. While I am not a fanatical gamer, I do game on my home system and have no complaints with Vista's performance, even with games that are not "Vista compatible". I do know some very high end gamers who have built custom Vista gaming rigs, and rave about their systems. Perhaps some day people will get over the fact that Vista is different and realize it is not something to fear.
      itpro_z
      • Yup.

        One thing I'm impressed with in Vista is the game support - all games I've used so far that were created for XP also work fine in Vista. It seems to me that using Vista has little, if any, impact on my games.

        . . . and now that I can finally see the exact figures used, what I've been saying all along has been true: They're all staying in the same ballpark. There are no vast frame rate differences. I frankly don't think the vast majority of gamers are going to notice or care if CoH runs at 28 fps instead of 30 fps.
        CobraA1
        • Just for the record ...

          ... my gaming PCs run Vista ...

          While you might be right in saying that this is fuss over nothing, don't you think that there's a valid point to be made that Microsoft's brand new OS is being outperformed, even a little bit, by an OS that's getting long in the tooth?
          Adrian Kingsley-Hughes
          • Only the point that has been tiredly made since DOS.

            Only the point that's tiredly made ever since DOS - as OSes advance, they add more features and overhead. Windows 98 outperformed Windows XP, and Windows 3.11 outperformed Windows 98.

            If you wanna really complain about OS overhead, slip your DOS disks in and watch your old DOS games go so fast you can't realistically play them without special software to slow them down.

            So no, there is no point to be made - except that Microsoft is more concerned about features than performance. We've known that for 17 years.
            CobraA1
          • Same can be said about games

            Games get new features and require bigger and better hardware to the point where it's hard to keep up. For me often I get a game that is sluggish and over time I upgrade and rebuild to the point where the game is working great, usually a year or so later.

            The OS is pretty much the same except the feature creep comes every few year instead of every new game.
            voska1
          • Whoa there ...

            ... None of the games I've listed there will run in DOS, but they all run on XP and Vista so that point is bogus. The issue here is taking one application and running it on two different platforms and getting different results.

            Msybre you're right, OSes are now about features and not performance and that we all should be using consoles for gaming. But looking today at Principaled Technologies benchmark of XP vs. Vista and the <http://www.principledtechnologies.com/Clients/Reports/Microsoft/microsoft.htm> reports try to make a couple of seconds gain here and there sound huge ... well, if those seconds add up, so do frame rates. If Microsoft didn't feel that way, why commission the study?
            Adrian Kingsley-Hughes
          • why MS commisioned the study?

            I'm guessing they were tired of hearing that Vista was dog slow compared to XP, you think? The Computerworld article on this study shows the two OS very close but Vista took the nod after a cold boot and XP after sleep, but the difference was very small in all cases. So much for Vista being demonstrably slow?
            eggmanbubbagee
          • Rather,

            it seems to me that MS is doing a pretty steady job of making the most of their features/performance tradeoffs, actually.
            Vista's feature improvements are far deeper than most bother to look or develop for outside of business use, and as the benchmarks continue to show, performance has held fairly steady for mainstream apps and functions across the latest OS's. The more the drivers/apps adjust to Vista specifically, the less performance disparity will show.

            Mountains out of molehills, all the way around.
            CraiGrrr
          • There's some exceptions to that rule, Cobra...

            [i]If you wanna really complain about OS overhead, slip your DOS disks in and watch your old DOS games go so fast you can't realistically play them without special software to slow them down.[/i]

            Doom/Doom 2 engine games for DOS and Windows are all frame-limited to 35 fps, no matter how fast your processor is.

            Early 3D accelerated games, such as any of the Star Wars games from around the time the second trilogy came out, will just run incredibly smooth. My copy of Pod Racer running on my Core 2 Duo e8400's Windows 98SE drive probably averages around 300 fps or so, and GLQuake 2 in Windows XP is nice too.

            Wolfenstein 3D (yes, you better remember it) is playable, but enemy AI is insane and the fact that framerates are so high really shows how far game engines have come in the last 16 years.
            nix_hed
    • I have to agree with CobraA1 here...

      I've noticed with Hellgate:London on my machine (x2 5000+, 4GB of RAM, X1950Pro video) that framerates are identical with DX9 with both XP and Vista 32. That being said, HG is actually faster under Vista when it comes to load times between instances. Vista's support for my old 17" CRT results in a crisper screen image as well, even though both OSes are driving the screen at the same refresh rate.

      It's not just ZDNet making a big deal - every so-called 'tech site' does. Game support forums are filled with people who've never used Vista who have an opinion about how bad it is, and how the only way to fix it is to put XP back on the machine. My machine would barely be considered a mid-range system by 1337 gamers, but it runs Vista very well, and plays every game I'm interested in superbly there. I have no complaints.
      neoanderthal