Vista SP1 - Benchmarking round-up

Given that I've made numerous posts relating to benchmarking Windows Vista Service Pack 1 (SP1) on this blog over the past few weeks I thought that many readers would appreciate a single post that pulled together all the individual posts and which drew together the thoughts, findings and conclusions from the individual posts.

• Vista SP1 Performance (this page)
• Vista SP1 vs. XP SP2
• Vista 32-bit vs. Vista 64-bit
• Office 2007 and Gaming Benchmarks
• Principled Technologies Vista SP1 vs. XP SP2 Reports
• Principled Technologies Vista SP1 vs. XP SP2 Reports

Vista SP1 Performance

I first got my hands on the Vista SP1 bits back on the 7th of February (install gallery here).  I received from Microsoft two discs, one containing the Vista SP1 upgrade pack for Vista 32-bit and Vista 64-bit, and a disc containing an updated Vista installation image which included SP1 (32-bit only).

I started off by installing SP1 onto one of my test systems, and very soon felt that SP1 was indeed a worthwhile install. 

Here's what I had to say on Day 1:

"I’m not ready to publish any benchmark test results yet, but on first blush SP1 feels snappier than Vista RTM. Manipulating files certainly feels a lot better and many of those awful (and workflow disrupting) lags are gone."

That same day I carried out some preliminary benchmarking, comparing Vista RTM to Vista SP1.  These test showed that Vista SP1 did, as Microsoft promised it would, improve file copy performance, as well as improving the way that Vista handled compressed files.  However, synthetic benchmark results (using PassMark PerformanceTest 6.1 and PCMark Vantage) showed a drop in performance.

By the 8th of February I'd carried out a number of system upgrades, including that of my main workstation, which was the first 64-bit SP1 upgrade I'd carried out.  I still didn't have any robust benchmark results but I did post my thoughts so far:

• Overall, the system feels quicker. There’s a snappy feeling to carrying out tasks (although this system was no slouch under Vista RTM!).
• Navigating network shares is a lot faster.
• Remote Desktop Connection is a lot better - I can connect to remote machines faster and the experience is noticeably better.
• There has been no change to the Windows Experience Index scale - 5.9 is still the top ceiling of the scale.
• Recovering from sleep is faster and the system is up and ready to use quicker.

By the 14th of February I'd completed an in-depth file-copy/compress/decompress/network transfer benchmark comparing Vista RTM with Vista SP1.  These benchmark results showed impressive performance gains for SP1 over a Vista install which didn't have SP1 installed.

"Credit where it’s due, Microsoft does seem to have eliminated a considerable number of Vista bottlenecks in SP1.  Two words sum up how I feel - I’m impressed."

Vista SP1 vs. XP SP2 -->

Vista SP1 vs. XP SP2

As soon as I'd published my Vista RTM vs. Vista SP1 benchmarks people started asking me about how Vista SP1 compares to XP SP2.  The upshot of this set of benchmark results was that XP SP2 walked away with the advantage - although drive to drive copy of a single file was much faster under Vista SP1 than under XP SP2.

Here's the conclusion I wrote to that piece:

So, onto conclusions.  Looking at the data there’s only one conclusion that can be drawn - Windows XP SP2 is faster than Windows Vista SP1.  End of story.  Out of the fifteen tests carried out, XP SP2 beat Vista SP1 in eleven, Vista SP1 beat XP SP2 in two of the tests, and two of the tests resulted in a draw.

The first Vista SP1 vs. XP SP2 benchmark results was quickly followed by a second set.  Here I tried to focus away from the raw results and try to find out which OS is the most responsive under load.  The conclusion to all this testing was as follows:

"So, what this long-winded series of tests shows is that heavy file copy operations has less of an effect on the overall responsiveness when running Vista SP1 than when running XP SP2 (on the test system, all things being equal)." 

Vista 32-bit vs. Vista 64-bit -->

Vista 32-bit vs. Vista 64-bit

In the next series of benchmarks I compared the performance of Vista 32-bit (RTM and SP1) and Vista 64-bit (RTM and SP1).  These benchmarks show that Vista 64-bit taking the lead over Vista 32-bit:

• SP1 causes a drop in PassMark Performance ratings on both Vista 32-bit and Vista 64-bit.
• The drop in the PassMark Performance ratings is greater for 64-bit Vista than it is for 32-bit Vista.
• However, both RTM and SP1 versions of Windows Vista 64-bit show a much higher PassMark Performance rating Windows Vista 32-bit.
• Boot times for Vista 64-bit are longer than Vista 32-bit.

These benchmarks were later augmented by a second set which used PCMark Vantage and CINEBENCH R10 benchmark tools. 

• In all the 64-bit tests (PCMark Vantage and CINEBENCH R10), Vista 64-bit RTM beat Vista 64-bit SP1.
• Vista 64-bit RTM also got the top spot on the PCMark Vantage 32-bit benchmark too, with Vista 32-bit RTM coming second, Vista 32-bit SP1 third and Vista 64-bit SP1 coming in forth.
• In most of the CINEBENCH R10 32-bit tests, Vista 32-bit RTM came in first and Vista 64-bit RTM came in second, Vista 32-bit SP1 came in third and Vista 64-bit SP1 came in third.
• All of the systems showed a decline in benchmark scores after installing SP1.

Office 2007 and Gaming Benchmarks -->

Office 2007 and Gaming Benchmarks

To round off my benchmarking I took a look at Office 2007 performance under Vista RTM and Vista SP1, along with gaming performance under the two operating systems. 

When examining the frame rate results from the gaming benchmarks, we noticed that there is little in the way of variation between XP SP2, Vista RTM and Vista SP1:

Here's the conclusion I came to for the Office 2007 benchmarking:

"Since we received very consistent results across the tests, it’s clear to me the impact that Vista Service Pack 1 has on the performance of Microsoft Office 2007 applications - almost none at all.  The 8 second variation in the average score between Vista RTM and Vista SP1 under no load is not going to be noticeable during normal usage.  This is good news for those who feared that SP1 might slow down Office applications, but bad news for those expecting a performance boost."

And here are the conclusions to the gaming benchmark marathon:

"So, what’s the bottom line?  Well, XP SP2 is still the better OS when it comes to gaming.  However, Vista is getting better.  Thanks to several performance updates from Microsoft, better drivers from hardware vendors and updates from the game makers themselves (a good example of this improvement is Oblivion, which was awful on Vista when it went RTM but has improved to the point where it’s actually better on Vista than on XP).  What would be interesting to try (although I don’t have the time to do this) would be to re-run the benchmarks with a bare, unpatched Vista RTM and compare this to Vista SP1 - my guess is that you’d see significant improvement.  Still, even comparing Vista SP1 to XP SP2, it’s clear that Vista has a long way to go before it can beat its older sibling."

Principled Technologies Vista SP1 vs. XP SP2 Reports -->

Principled Technologies Vista SP1 vs. XP SP2 Reports

I wasn't the only person working on benchmarking Vista SP1 and comparing the results to XP SP2.  A set of reports released by Principled Technologies enjoyed a fair amount of press.  I examined these Microsoft commissioned reports and came to the following conclusions:

"These test results are remarkably similar to those I came up with.  For example, when I tested the responsiveness of Office 2007 using OfficeBench I concluded that there’s really little difference between Vista RTM and Vista SP1.  When Principled Technologies are talking about differences of halves and quarters of seconds, that’s just another way of saying that there’s really little difference.

"Another point worth noting from the reports is that when Vista RTM and Vista SP1 are compared, there are hardly any real improvements to be seen.  Given that SP1 really does speed up operations such as file copy, these don’t really have much of an effect on real world usage.  Again, this aligns with what I’ve seen during testing.  Sure, if you look for the changes, they are there, but in normal use you don’t see the effect that much at all."

However, despite seeing a correlation between the results that I published and those published by Principled Technologies, two aspects relating to the benchmarking process bothered me at the time:

• First, these tests are an example of closed-source benchmarking.  The tests use a Microsoft tool that's used internally and not available widely (ScenCap) for script creation, and Principled Technologies have not released the test scripts that were used in the report.  Because of this there is no way for me or anyone else to replicate the tests carried.  Without open access to the tools used for testing, the numbers might be dead-on accurate or outputted by a random number generator - there's no way of knowing.
• Another problem is that both reports contain an appendix which covers an HP a1320y desktop system.  According to the reports the results from this system were omitted because it “produced unacceptably high variability in its test results.”  However, a quick run through the scores provided would seem to indicate that XP beat Vista RTM and SP1 on many of the tests that were carried out. 

I've approached Principled Technologies for clarification on these points and so far have not received any satisfactory answers.

OK, so what's the bottom line? -->

OK, so what's the bottom line?

OK, so what's the bottom line you should take away from all this benchmarking?  Here, summarized into as few a bullet points as possible, are my thoughts:

• Without a doubt, Vista SP1 improves on Vista RTM in the following areas: file copy, compress/decompress, and network transfer.
• Both real-world and synthetic benchmarking that I've carried out seems to show that XP SP2 still holds an edge over Vista SP1.
• Vista SP1 shows a small but consistent drop in synthetic benchmark results compared to Vista RTM.
• Vista SP1 performs better under load and appears to be more responsive than Vista RTM and XP SP2.
• In gaming and Office 2007 benchmark tests, the differences between XP SP2, Vista RTM and Vista SP1 seem to be negligible.

So, what should you do?

• If you're already running Vista RTM, I see no real reason not to install SP1 when it's released.  Synthetic benchmark results might take a small hit but overall system responsiveness under load is improved. 
• If you're running XP SP2, things are far less clear cut.  If you are looking to make the move to Vista, then you're benefiting from a far better, faster, more robust Vista than people who upgraded a year ago.  In fact, you'll be getting Vista for almost no performance hit.  So, if you're looking for the features that Vista has to offer, it's a good time to make the move (especially since Microsoft has cut the price of Vista upgrades).  However, if Aero, changes to the OS security and DirectX 10 aren't important to you, then there's no reason why you can't stick with XP for another year or so - Vista is still far from being mainstream.

Thoughts?

<< Home >>