XP SP2 vs. Vista RTM vs. Vista SP1: Gaming benchmark

Here are the benchmark results that many of you have been waiting for - a look at how well ten popular games work on XP SP2, Vista RTM and Vista SP1.

It's been nearly a year since I took a look at the state of gaming under the Vista OS.  The last time I compared XP SP2 to Vista, I was disappointed with Microsoft's new OS as a gaming platform because I could get far more bang for my bucks out of XP than I could out of the new OS.  Partly I put this down to immature drivers, but on the whole I was convinced that at the core of the problem was Vista. 

So, is Vista worth bothering with as a gaming platform?

The tests

The tests will be carried out on the AMD Spider platform that I have set up in the lab (Phenom 9700, Radeon 3850 graphics card, 2GB of RAM …).  I’ve used this system as the platform for a number of benchmarks I’ve run over the past week (for a full spec, see this post).

Here's the deal.  I set up three images for the system - one based on XP SP2, one on Vista RTM and one on Vista SP1 (all 32-bit flavor).  On each of these images all the updates provided by Microsoft were installed.  The installation of drivers in addition to those supplied by Microsoft was kept to a minimum (XP SP2 required more drivers than Vista).  On each of the images the latest ATi Catalyst drivers (version 8.2) were installed.  Then, in order to be able to measure and record frame rates, the latest version of the FRAPS utility was installed.

Finally, the games we were going to were then installed and each one updated to the latest version.  Here's a list of the games used:

• BioShock
• Call of Duty 4
• Company of Heroes
• Crysis
• Doom 3
• F.E.A.R.
• The Elder Scrolls IV: Oblivion
• Supreme Commander
• Unreal Tournament 3
• World in Conflict

For these tests we chose not to rely on any of the canned benchmark tools supplied with some of these games.  Instead we relied on real-world play - that is, playing the same bit of the game on each of the platforms (a technique that HardOCP use when benchmarking).  While this method is subject to variations, if you've played games for long enough and are familiar with the map, you can pull this off quite well.

Note: Why not used canned benchmarks?  Well, by not doing so we are ruling out the possibility that the graphics card drivers that we use (or even the OS) has in any way been optimized to produce better results for the canned benchmarks that you might see in the game. 

To come up with workable graphics settings we first experimented with each of the games in turn.  When we found graphics settings that worked well (all the tests were run at 1280 x 1024 resolution in full-screen mode with the games set to either medium or high quality graphics) these settings were carefully recorded replicated them across the three platforms.

So to come up with the average frame rate (measured in frames per second) we play the same bit in each game with FRAPS recording, repeat this four times.  If there was any wild variation in the data we went back and took another stab at it (we only had to do this once, with Supreme Commander).  Once we were happy with the data we averaged it out to get a final frame rate score. 

We were very pleased with the end data because we ended up with runs where the average frame rate only varied by one or two frames per second.  This gave us confidence in the final results we achieved.

The results -->

The results

These tests took longer to carry out than I'd planned - in fact, the results that follow are the fruits of over a week of work.  Still, to get a hands-on snapshot of the current state of gaming on both XP SP2, Vista RTM and Vista SP1, it was worth the time and effort.

Here are the results. 

The table below shows the average frame rates for each game on each platform.  We also show the percentage difference compared to XP SP2 (so, for example, a value of -10% indicates a drop in frame rate of 10% compared to XP SP2, which 10% would indicate a 10% increase in frame rate compared to XP SP2).

Here are graphs showing average frame rates for the ten games across all three platforms:

Conclusions -->

Conclusions

Because we're juggling with a lot of data here, I'm going to try to present the conclusions as clearly and concisely as possible.

• On the whole, applying SP1 to Vista makes little difference to most games.
• Seven out of ten of the games tested produced frame rate averages which were lower under Vista than XP SP2.
• However, a closer look at the numbers shows that when Vista is slower than XP SP2, the actual frame rate differences are only in the single digits.
• The only exception to the above is Doom 3, which shows a drop of 19 frames per second on Vista RTM and 15 frames per second on Vista SP1.  This is the largest variation we saw during testing.
• The three games which performed better on Vista than on XP SP2 were Call of Duty 4, F.E.A.R. and The Elder Scrolls IV: Oblivion.
• Vista gave The Elder Scrolls IV: Oblivion the best performance boost compared to XP SP2, working out at an average of 9 frames per second faster on Vista RTM and 13 frames per second faster on Vista SP1.

 So, what's the bottom line?  Well, XP SP2 is still the better OS when it comes to gaming.  However, Vista is getting better.  Thanks to several performance updates from Microsoft, better drivers from hardware vendors and updates from the game makers themselves (a good example of this improvement is Oblivion, which was awful on Vista when it went RTM but has improved to the point where it's actually better on Vista than on XP).  What would be interesting to try (although I don't have the time to do this) would be to re-run the benchmarks with a bare, unpatched Vista RTM and compare this to Vista SP1 - my guess is that you'd see significant improvement.  Still, even comparing Vista SP1 to XP SP2, it's clear that Vista has a long way to go before it can beat its older sibling.

Still, the fact that some gamers work better on Vista means that those after the maximum frame rates possible might want to set up dual boot systems.

As always, your mileage WILL vary!

Thoughts?

Note: Here are links to previous Vista benchmark posts: 1 - Some systems showing incredible SP1 performance boost | 2 - Vista SP1 vs. XP SP2 - Benchmarked | 3 - Vista SP1 vs. XP SP2 - Part Deux | 4 - Vista 32-bit vs. Vista 64-bit - Benchmarked | 5 - Vista: 32-bit vs. 64-bit & RTM vs. SP1 | 6 - Vista RTM vs. Vista SP1 - Office 2007 benchmarking.

<< Home >>